High-speed event detection using a compressive-sensing hyperspectral-imaging architecture

ABSTRACT

A compressive imaging system and method for quickly detecting spectrally and spatially localized events (such as explosions or gun discharges) occurring within the field of view. An incident light stream is modulated with a temporal sequence of spatial patterns. The wavelength components in the modulated light stream are spatially separated, e.g., using a diffractive element. An array of photodetectors is used to convert subsets of the wavelength components into respective signals. An image representing the field of view may be reconstructed based on samples from some or all the signals. A selected subset of the signals are monitored to detect event occurrences, e.g., by detecting sudden changes in intensity. When the event is detected, sample data from the selected subset of signals may be analyzed to determine the event location within the field of view. The event location may be highlighted in an image being generated by the imaging system.

RELATED APPLICATION DATA

This application claims the benefit of priority to U.S. ProvisionalApplication No. 61/553,347, filed on Oct. 31, 2011, entitled “Hi-SpeedCompressive Sensing Hyperspectral Imaging System”, invented by Robert F.Bridge and Lenore McMackin, which is hereby incorporated by reference inits entirety as though fully and completely set forth herein.

Furthermore, this application is a continuation-in-part of U.S. patentapplication Ser. No. 13/534,528, filed on Jun. 27, 2012, entitled“Mechanisms for Conserving Power in a Compressive Imaging System”,invented by Bridge, Tidman, McMackin and Chatterjee, which claims thebenefit of priority to U.S. Provisional Application No. 61/502,153,filed on Jun. 28, 2011, entitled “Various Compressive SensingMechanisms”, invented by Tidman, Weston, Bridge, McMackin, Chatterjee,Woods, Baraniuk and Kelly. patent application Ser. No. 13/534,528 andProvisional Application 61/502,153 are hereby incorporated by referencein their entireties as though fully and completely set forth herein.

FIELD OF THE INVENTION

The present invention relates to the field of compressive imaging, andmore particularly, to mechanisms for detecting spectrally and spatiallylocalized events using a compressive sensing hyperspectral imager.

DESCRIPTION OF THE RELATED ART

According to Nyquist theory, a signal x(t) whose signal energy issupported on the frequency interval [−B,B] may be reconstructed fromsamples {x(nT)} of the signal x(t), provided the rate f_(s)=1/T_(s) atwhich the samples are captured is sufficiently high, i.e., provided thatf_(s) is greater than 2B. Similarly, for a signal whose signal energy issupported on the frequency interval [A,B], the signal may bereconstructed from samples captured with sample rate greater than B-A. Afundamental problem with any attempt to capture a signal x(t) accordingto Nyquist theory is the large number of samples that are generated,especially when B (or B-A) is large. The large number of samples istaxing on memory resources and on the capacity of transmission channels.

Nyquist theory is not limited to functions of time. Indeed, Nyquisttheory applies more generally to any function of one or more realvariables. For example, Nyquist theory applies to functions of twospatial variables such as images, to functions of time and two spatialvariables such as video, and to the functions used in multispectralimaging, hyperspectral imaging, medical imaging and a wide variety ofother applications. In the case of an image I(x,y) that depends onspatial variables x and y, the image may be reconstructed from samplesof the image, provided the samples are captured with sufficiently highspatial density. For example, given samples {I(nΔx,mΔy)} captured alonga rectangular grid, the horizontal and vertical densities 1/Δx and 1/Δyshould be respectively greater than 2B_(x) and 2B_(y), where B_(X) andB_(y) are the highest x and y spatial frequencies occurring in the imageI(x,y). The same problem of overwhelming data volume is experienced whenattempting to capture an image according to Nyquist theory. The modemtheory of compressive sensing is directed to such problems.

Compressive sensing relies on the observation that many signals (e.g.,images) of practical interest are not only band-limited but also sparseor approximately sparse when represented using an appropriate choice oftransformation, for example, a transformation such as a Fouriertransform, a wavelet transform or a discrete cosine transform (DCT). Asignal vector v is said to be K-sparse with respect to a giventransformation T when the transformation of the signal vector, Tv, hasno more than K non-zero coefficients. A signal vector v is said to besparse with respect to a given transformation T when it is K-sparse withrespect to that transformation for some integer K much smaller than thenumber L of components in the transformation vector Tv.

A signal vector v is said to be approximately K-sparse with respect to agiven transformation T when the coefficients of the transformationvector, Tv, are dominated by the K largest coefficients (i.e., largestin the sense of magnitude or absolute value). In other words, if the Klargest coefficients account for a high percentage of the energy in theentire set of coefficients, then the signal vector v is approximatelyK-sparse with respect to transformation T. A signal vector v is said tobe approximately sparse with respect to a given transformation T when itis approximately K-sparse with respect to the transformation T for someinteger K much less than the number L of components in thetransformation vector Tv.

Given a sensing device that captures images with N samples per image andin conformity to the Nyquist condition on spatial rates, it is often thecase that there exists some transformation and some integer K very muchsmaller than N such that the transform of each captured image will beapproximately K sparse. The set of K dominant coefficients may vary fromone image to the next. Furthermore, the value of K and the selection ofthe transformation may vary from one context (e.g., imaging application)to the next. Examples of typical transforms that might work in differentcontexts include the Fourier transform, the wavelet transform, the DCT,the Gabor transform, etc.

Compressive sensing specifies a way of operating on the N samples of animage so as to generate a much smaller set of samples from which the Nsamples may be reconstructed, given knowledge of the transform underwhich the image is sparse (or approximately sparse). In particular,compressive sensing invites one to think of the N samples as a vector vin an N-dimensional space and to imagine projecting the vector v ontoeach vector in a series of M vectors {R(i): i=1, 2, . . . , M} in theN-dimensional space, where M is larger than K but still much smallerthan N. Each projection gives a corresponding real number s(i), e.g.,according to the expressions(i)=<v,R(i)>,where the notation <v,R(i)> represents the inner product (or dotproduct) of the vector v and the vector R(i). Thus, the series of Mprojections gives a vector U including M real numbers: U_(i)=s(i).Compressive sensing theory further prescribes methods for reconstructing(or estimating) the vector v of N samples from the vector U of M realnumbers. For example, according to one method, one should determine thevector x that has the smallest length (in the sense of the L₁ norm)subject to the condition that ΦTx=U, where Φ is a matrix whose rows arethe transposes of the vectors R(i), where T is the transformation underwhich the image is K sparse or approximately K sparse.

Compressive sensing is important because, among other reasons, it allowsreconstruction of an image based on M measurements instead of the muchlarger number of measurements N recommended by Nyquist theory. Thus, forexample, a compressive sensing camera would be able to capture asignificantly larger number of images for a given size of image store,and/or, transmit a significantly larger number of images per unit timethrough a communication channel of given capacity.

As mentioned above, compressive sensing operates by projecting the imagevector v onto a series of M vectors. As discussed in U.S. Pat. No.8,199,244 (issued Jun. 12, 2012, invented by Baraniuk et al.) andillustrated in FIG. 1, an imaging device (e.g., camera) may beconfigured to take advantage of the compressive sensing paradigm byusing a digital micromirror device (DMD) 40. An incident lightfield 10passes through a lens 20 and then interacts with the DMD 40. The DMDincludes a two-dimensional array of micromirrors, each of which isconfigured to independently and controllably switch between twoorientation states. Each micromirror reflects a corresponding portion ofthe incident light field based on its instantaneous orientation. Anymicromirrors in a first of the two orientation states will reflect theircorresponding light portions so that they pass through lens 50. Anymicromirrors in a second of the two orientation states will reflecttheir corresponding light portions away from lens 50. Lens 50 serves toconcentrate the light portions from micromirrors in the firstorientation state onto a photodiode (or photodetector) situated atlocation 60. Thus, the photodiode generates a signal whose amplitude atany given time represents a sum of the intensities of the light portionsfrom the micromirrors in the first orientation state.

The compressive sensing is implemented by driving the orientations ofthe micromirrors through a series of spatial patterns. Each spatialpattern specifies an orientation state for each of the micromirrors. Theoutput signal of the photodiode is digitized by an A/D converter 70. Inthis fashion, the imaging device is able to capture a series ofmeasurements {s(i)} that represent inner products (dot products) betweenthe incident light field and the series of spatial patterns withoutfirst acquiring the incident light field as a pixelized digital image.The incident light field corresponds to the vector v of the discussionabove, and the spatial patterns correspond to the vectors R(i) of thediscussion above.

The incident light field may be modeled by a function I(x,y,t) of twospatial variables and time. Assuming for the sake of discussion that theDMD comprises a rectangular array, the DMD implements a spatialmodulation of the incident light field so that the light field leavingthe DMD in the direction of the lens 50 might be modeled by{I(nΔx,mΔy,t)*M(n,m,t)}where m and n are integer indices, where I(nΔx,mΔy,t) represents theportion of the light field that is incident upon that (n,m)^(th) mirrorof the DMD at time t. The function M(n,m,t) represents the orientationof the (n,m)^(th) mirror of the DMD at time t. At sampling times, thefunction M(n,m,t) equals one or zero, depending on the state of thedigital control signal that controls the (n,m)^(th) mirror. Thecondition M(n,m,t)=1 corresponds to the orientation state that reflectsonto the path that leads to the lens 50. The condition M(n,m,t)=0corresponds to the orientation state that reflects away from the lens50.

The lens 50 concentrates the spatially-modulated light field{I(nΔx,mΔy,t)*M(n,m,t)}onto a light sensitive surface of the photodiode. Thus, the lens and thephotodiode together implement a spatial summation of the light portionsin the spatially-modulated light field:

${S(t)} = {\sum\limits_{n,m}^{\;}{{I\left( {{n\;\Delta\; x},{m\;\Delta\; y},t} \right)}{{M\left( {n,m,t} \right)}.}}}$

Signal S(t) may be interpreted as the intensity at time t of theconcentrated spot of light impinging upon the light sensing surface ofthe photodiode. The A/D converter captures measurements of S(t). In thisfashion, the compressive sensing camera optically computes an innerproduct of the incident light field with each spatial pattern imposed onthe mirrors. The multiplication portion of the inner product isimplemented by the mirrors of the DMD. The summation portion of theinner product is implemented by the concentrating action of the lens andalso the integrating action of the photodiode.

In U.S. Pat. No. 8,199,244, Baraniuk at el. teach that, “Many possibleembodiments exist for full-color implementation” of the compressivesensing camera, “including a series of prisms to separate the signalbetween 3 separate photodiodes. In a similar manner we can easily extendthe capabilities of our camera for more detailed multispectral orhyperspectral imaging.” Thus, Baranuik et al. suggest the possibility ofperforming hyperspectral imaging using a compressive sensing device. Oneproblem that exists generally in the field of hyperspectral imaging isthe problem of detecting events such as explosions, gun discharges andchemical reactions that occur suddenly and are concentrated in a limitedspectral band, e.g., a band significantly smaller than the range ofwavelengths being captured by the hyperspectral imager. It would be slowand inefficient to monitor the entire wavelength range to detect suchevents. A compressive sensing implementation of hyperspectral imagingwould be subject to the same problem. It would be slow and inefficientto continuously reconstruct images at each wavelength of thehyperspectral range in order to detect spectrally-limited events. Thus,there exists a need for compressed sensing architectures capable ofquickly and efficiently detecting such events. Furthermore, havingdetected such an event, it would be desirable to quickly locate thespatial source of the event within the field of view.

SUMMARY

In one set of embodiments, a system may be configured to acquirecompressive imaging measurements at each of a plurality of wavelengthbands within a wavelength spectrum. The measurements at each wavelengthband may be used to reconstruct a corresponding image. Thus, the systemmay realize a multispectral or hyperspectral imager. The system may befurther configured to detect and locate a spectral event occurringwithin the field of view by monitoring and analyzing thecompressively-acquired measurements corresponding to a selected subsetof the wavelength bands. For example, it may be known beforehand thatcertain types of explosion or gun discharge or chemical reaction willexhibit high-intensity radiation in a given range of wavelengths. Thus,the system may selectively monitor that range of wavelengths to detectsuch events. Other types of event may express in different wavelengthranges. Thus, the range of wavelengths to be monitored by the system maybe programmable.

The system may include a spectral separation subsystem, an array oflight sensing elements, a sampling subsystem and a detection unit. Thespectral separation subsystem may be configured to receive a modulatedlight stream, where the modulated light stream is generated bymodulating an incident light stream with a temporal sequence of spatialpatterns. The spatial patterns may be measurement patterns, i.e., may beincoherent relative to the set of patterns in which image (or imagesequence) carried by the incident light stream is sparse orcompressible. (Thus, the samples acquired by the system at eachwavelength band comprises compressive measurements.) The spectralseparation subsystem is configured to separate the modulated lightstream into a plurality of wavelength components, e.g., using adiffraction grating or a prism or a series of spectral filters, etc. Theplurality of wavelength components may comprise a continuum ofwavelength components spread out spatially. Alternatively, the pluralityof wavelength components may include wavelength components that arespatially isolated, e.g., in terms of discrete beams.

The light sensing elements (e.g., photodiodes) may be configured toreceive respective subsets (e.g., bands) of the wavelength componentsand to generate respective signals. Each of the signals representsintensity of the respective subset of the wavelength components as afunction of time. The sampling subsystem may be configured to sample thesignals in order to obtain respective sample sequences. For example, thesampling system may comprise an array of analog-to-digital converters.

The detection unit may be configured to monitor a selected subset of thesignals to detect an event occurring within a field of viewcorresponding to the incident light stream. The action of detecting theevent may include determining when the selected subset of signalssatisfy a pre-determined signal condition. For example, thepre-determined signal condition may be the condition that the signals ofthe selected subset simultaneously exceed respective programmablethresholds. As another example, the pre-determined signal condition maybe the condition that rates of change of the respective signals of theselected subset simultaneously exceed respective programmable ratethresholds. As yet another example, the pre-determined signal conditionmay be the logical AND of the two example conditions given above. As yetanother example, the pre-determined signal condition is the conditionthat the signals of the selected subset have respective values thatconform to a pre-determined spectral signature.

In some embodiments, the system may also include a processing unitconfigured to reconstruct a temporal sequence of images based on thesample sequences acquired by the sampling subsystem, i.e., more thanjust the selected subset of sample sequences. Each image of the temporalsequence may be reconstructed based on a corresponding temporal windowof sample data. (Successive ones of the temporal windows may overlap intime by a pre-determined amount.) The images of the temporal sequencemay be interpreted as broad spectrum images or full-spectrum images. Theimages of the temporal sequence may be displayed via a display device.

In some embodiments, the processing unit may be configured toreconstruct a first image and a second image in response to thedetection of the event. The first image may be reconstructed based on afirst window of samples taken from the sample sequences corresponding tothe selected subset. The second image may be reconstructed based on asecond window of samples taken from the sample sequences correspondingto the selected subset. (The first image and the second image may beinterpreted as partial-spectrum images since they do not incorporatewavelength components outside the selected subset.) The first windowcorresponds to a first time interval prior to the event while the secondwindow corresponds to a second time interval that at least partiallyincludes the event. After reconstructing the first image and the secondimage, the processing unit may determine spatial localizationinformation based on a difference between the first image and the secondimage. (The spatial localization information indicates where the eventhas occurred in the field of view.) The processing unit may inject avisual representation of the spatial localization information into atleast a subset of the images of the temporal sequence of images. Thus,the user is given a visual cue as to the location of the event. The usermay more readily interpret and understand the event by being able toquickly focus his/her gaze upon the spatial neighborhood of the event asit is occurring within the general scene context. Thus, the user may bebetter prepared to assert counter measures and/or take evasive action.

In some embodiments, the processing unit may be configured toreconstruct a partial-spectrum image in response to the detection of theevent. The partial-spectrum image may be based on a window of samplestaken from the sample sequences corresponding to the selected subset.The window corresponds to a first time interval that at least partiallyincludes the event. The processing unit may determine spatiallocalization information based on the first image, where the spatiallocalization information indicates where the event has occurred in thefield of view. The processing unit may then inject a visualrepresentation of the spatial localization information into at least asubset of the images of said temporal sequence of images.

In some embodiments, the processing unit may be configured to perform asearch process in response to the detection of the event. The searchprocess may operate on one or more of the sample sequences belonging tothe selected subset and during the occurrence of the event, in order toidentify a spatial subregion within the field of view that contains theevent. The search process may include injecting search patterns into thetemporal sequence of spatial patterns, and analyzing the samples of theone or more sample sequences in response to the injection of the searchpatterns. In one embodiment, the search process may include ahierarchical search based on a quadtree, where the quadtree correspondsto a recursive partitioning of the field of view into rectangularsubsets.

In one set of embodiments, a compressive-sensing hyperspectral-imagingsystem may include a digital micromirror device, a spectral separationsubsystem, a first array of light sensing elements, a first samplingsubsystem and a detection unit.

The digital micromirror device (DMD) may be configured to receive anincident light stream, and modulate the incident light stream with atemporal sequence of spatial patterns to obtain a modulated light streamand a complementary modulated light stream.

The spectral separation subsystem may be configured to receive themodulated light stream, and separate the modulated light stream into aplurality of wavelength components.

The light sensing elements of the first array may be configured toreceive respective subsets of the wavelength components and to generaterespective spectral element signals. Each of the spectral elementsignals represents intensity of the respective subset of the wavelengthcomponents as a function of time. The first sampling subsystem may beconfigured to sample the spectral element signals in order to obtainrespective spectrally-limited sample sequences.

The detection unit may be configured to monitor a selected subset of thespectral element signals to detect an event occurring within a field ofview corresponding to the incident light stream. The action of detectingthe event includes determining when the selected subset of the spectralelement signals satisfy a pre-determined signal condition.

In some embodiments, the system may also include a second array of lightsensing elements. The light sensing elements of the second array may beconfigured to convert respective spatial portions of the complementarymodulated light stream into respective spatial element signals. A secondsampling subsystem may be used to sample the spatial element signals toobtain respective spatially-limited sample sequences. A processing unitmay be used to reconstruct a temporal sequence of images based on thespatially-limited sample sequences.

In alternative embodiments, the system may also include a light sensingdevice configured to convert the complementary modulated light streaminto a device output signal representing intensity of the complementarymodulated light stream as a function of time. An analog-to-digitalconverter (ADC) may be used to sample the device output signal to obtainan output sample sequence. A processing unit may be used to reconstructa temporal sequence of images based on the output sample sequence.

In one set of embodiments, a compressive-sensing hyperspectral-imagingsystem may include a light modulation unit, a diffraction unit, a firstarray of light sensing elements, a first sampling subsystem and adetection unit.

The light modulation unit may be configured to receive an incident lightstream, and modulate the incident light stream with a temporal sequenceof spatial patterns to obtain a modulated light stream.

The diffraction unit may be configured to diffract the modulated lightstream into a zeroth-order beam and a first-order beam, where thefirst-order beam includes a plurality of wavelength components that areseparated spatially.

The light sensing elements of the first array may be configured toreceive respective subsets of the wavelength components of thefirst-order beam and to generate respective spectral element signals.Each of the spectral element signals represents intensity of therespective subset of the wavelength components as a function of time.The first sampling subsystem may be configured to sample the spectralelement signals in order to obtain respective spectrally-limited samplesequences.

The detection unit may be configured to monitor a selected subset of thespectral element signals to detect an event occurring within a field ofview corresponding to the incident light stream. The action of detectingthe event includes determining when the selected subset of the spectralelement signals satisfy a pre-determined signal condition.

In some embodiments, the system also includes a second array of lightsensing elements, where the light sensing elements of the second arrayare configured to convert spatial portions of the zeroth-order beam intorespective spatial element signals. Each of the spatial element signalsrepresents intensity of the respective spatial portion as a function oftime. A second sampling subsystem may sample the spatial element signalsto obtain respective spatially-limited sample sequences. A processingunit may reconstruct a temporal sequence of images based on thespatially-limited sample sequences.

In alternative embodiments, the system also include a light sensingdevice configured to convert the zeroth-order beam into a device outputsignal representing intensity of the zeroth-order beam as a function oftime. An analog-to-digital converter (ADC) may be employed to sample thedevice output signal in order to obtain an output sample sequence. Aprocessing unit may reconstruct a temporal sequence of images based onthe output sample sequence.

Various additional embodiments are described in U.S. ProvisionalApplication No. 61/553,347, filed on Oct. 31, 2011, entitled “Hi-SpeedCompressive Sensing Hyperspectral Imaging System”, invented by Robert F.Bridge and Lenore McMackin.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiments isconsidered in conjunction with the following drawings.

FIG. 1 illustrates a compressive sensing camera according to the priorart.

FIG. 2A illustrates one embodiment of a system 100 that is operable tocapture compressive imaging samples and also samples of background lightlevel. (LMU is an acronym for “light modulation unit”. MLS is an acronymfor “modulated light stream”. LSD is an acronym for “light sensingdevice”.)

FIG. 2B illustrates an embodiment of system 100 that includes aprocessing unit 150.

FIG. 2C illustrates an embodiment of system 100 that includes an opticalsubsystem 105 to focus received light L onto the light modulation unit110.

FIG. 2D illustrates an embodiment of system 100 that includes an opticalsubsystem 117 to direct or focus or concentrate the modulated lightstream MLS onto the light sensing device 130.

FIG. 2E illustrates an embodiment where the optical subsystem 117 isrealized by a lens 117L.

FIG. 2F illustrates an embodiment of system 100 that includes a controlunit that is configured to supply a series of spatial patterns to thelight modulation unit 110.

FIG. 3A illustrates system 200, where the light modulation unit 110 isrealized by a plurality of mirrors (collectively referenced by label110M).

FIG. 3B shows an embodiment of system 200 that includes the processingunit 150.

FIG. 4 shows an embodiment of system 200 that includes the opticalsubsystem 117 to direct or focus or concentrate the modulated lightstream MLS onto the light sensing device 130.

FIG. 5A shows an embodiment of system 200 where the optical subsystem117 is realized by the lens 117L.

FIG. 5B shows an embodiment of system 200 where the optical subsystem117 is realized by a mirror 117M and lens 117L in series.

FIG. 5C shows another embodiment of system 200 that includes a TIR prismpair 107. (TIR is an acronym for Total Internal Reflection.)

FIG. 6 illustrates one embodiment of a system 600 that operates in a lowpower mode until a light variation event is detected, whereupon it turnson power to the light modulation unit 620.

FIG. 7 illustrates an embodiment of system 600 that includes atransmitter 660.

FIG. 7B illustrates an embodiment where system 600 transmitscompressively-acquired measurements to a remote system for remote image(or image sequence) reconstruction.

FIG. 8 illustrates an embodiment of system 600 where the light sensingdevice 640 is realized by a motion sensor.

FIG. 9 illustrates an embodiment of system 600 that includes a motionsensor in addition to the light sensing devices 630 and 640.

FIG. 10 illustrates an embodiment of system 600 where the opticalsubsystem 610 includes a TIR prism pair 610T.

FIG. 11 illustrates an embodiment of system 600 where the opticalsubsystem 610 includes a beam splitter 610S and a TIR prism pair 610T.

FIG. 12 illustrates an embodiment of system 600 including two lightsensing devices downstream from the light modulation unit 620.

FIG. 13 illustrates one embodiment of a system 1300 for detectingspectral events using compressively-acquired signal information from aselected subset of the spectral channels sensed by a light sensingarray.

FIG. 14 illustrates an embodiment of system 1300 where the detectionunit operates in the digital domain on digitized sample streams.

FIGS. 15A and 15B illustrates embodiments of system 1300 including aprocessing unit for performing operations such as image reconstruction,determination of event locations, etc.

FIG. 16 illustrates an example of a reconstructed image integrated witha visual indication of the location of event occurring in the field ofview.

FIG. 17 illustrates one embodiment of a method for detecting spectralevents using a selected subset of the spectral channel signals acquiredby a compressive-imaging device.

FIG. 18 illustrates one embodiment of a system 1800 for detectingspectral events, involving the use of a diffraction grating 1815 and amulti-channel sensing and detection unit 1825.

FIG. 19 illustrates one embodiment of the sensing and detection unit1825, including an array of photodiodes and an array of photodiodemonitoring blocks.

FIG. 20 illustrates one embodiment of the photodiode monitoring block(PMB), including an analog comparator circuit 2015.

FIG. 21 illustrates another embodiment of the photodiode monitoringblock (PMB), including a digital comparator unit DCU.

FIG. 22 illustrates one embodiment of an event detection and locationmethod performed, e.g., by the system controller block (SCB) of FIG. 19.

FIG. 23 illustrates one embodiment of an alternative event detection andlocation method performed, e.g., by the system controller block (SCB) ofFIG. 19.

FIG. 24 illustrates one embodiment of a system 2400 for performing eventdetection and/or hyperspectral imaging using a digital micromirrordevice (DMD).

FIG. 25 illustrates an embodiment of system 2400 that uses both theoutput light streams produced by the DMD. The second output stream(i.e., the complementary modulated light stream CMLS) is sensed with anarray 2515 of light sensing elements.

FIG. 26 illustrates another embodiment of system 2400 that uses both theoutput light streams produced by the DMD. The second output stream(i.e., the complementary modulated light stream CMLS) is sensed by alight sensing device 2615 (e.g., a photodiode).

FIG. 27 illustrates one embodiment of a dual TIR prism that may be usedto decrease the amount of space required to effectively separate themodulated light stream MLS and the complementary modulated light streamCMLS from each other and from the incident light stream, so that the twostreams may be separately sensed.

FIG. 28 illustrates one embodiment of a system 2800 including a dual TIRprism 2810.

FIG. 29 illustrates one embodiment of a system 2900 that performs eventdetection using a first-order diffraction beam generated by adiffraction unit.

FIG. 30 illustrates an embodiment of system 2900 that separately sensesthe zeroth-order diffraction beam generated by the diffraction unit.

FIG. 31 illustrates an embodiment of system 2900 where the diffractionunit is realized by a diffraction grating.

FIG. 32 illustrates an embodiment of system 2900 where the zeroth-orderdiffraction beam is sensed by a light sensing device (e.g., aphotodiode).

FIG. 33 illustrates one embodiment of a TIR prism pair 3300 configuredpartially transmit and partially reflect an input light stream ILS froman internal interface 3310.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Incorporations by Reference

The following patent applications are hereby incorporated by referencein their entireties as though fully and completely set forth herein.

U.S. patent application Ser. No. 13/631,626, filed on Sep. 28, 2012,entitled “Adaptive Search for Atypical Regions in Incident Light Fieldand Spectral Classification of Light in the Atypical Regions”, inventedby Richard G. Baraniuk and Tyler H. Weston;

U.S. patent application Ser. No. 13/534,414, filed on Jun. 27, 2012,entitled “Image Sequence Reconstruction based on Overlapping MeasurementSubsets”, invented by James M. Tidman, Tyler H. Weston, Matthew A.Herman and Lenore McMackin;

U.S. patent application Ser. No. 13/534,528, filed on Jun. 27, 2012,entitled “Mechanisms for Conserving Power in a Compressive ImagingSystem”, invented by Robert F. Bridge, James M. Tidman, Lenore McMackinand Sujoy Chatterjee;

U.S. patent application Ser. No. 13/534,249, filed on Jun. 27, 2012,entitled “User Control of the Visual Performance of a CompressiveImaging System”, invented by Robert F. Bridge, Donna E. Hewitt and TylerH. Weston;

U.S. Provisional Application No. 61/553,347, filed on Oct. 31, 2011,entitled “Hi-Speed Compressive Sensing Hyperspectral Imaging System”,invented by Robert F. Bridge and Lenore McMackin;

U.S. patent application Ser. No. 13/207,900, filed on Aug. 11, 2011,entitled “TIR Prism to Separate Incident Light and Modulated Light inCompressive Imaging Device”, invented by Lenore McMackin and SujoyChatterjee;

U.S. patent application Ser. No. 13/207,276, filed on Aug. 10, 2011,entitled “Dynamic Range Optimization in a Compressive Imaging System”,invented by Kevin F. Kelly, Gary L. Woods, Lenore McMackin, Robert F.Bridge, James M. Tidman and Donna E. Hewitt”;

U.S. patent application Ser. No. 13/197,304, filed on Aug. 3, 2011,entitled “Decreasing Image Acquisition Time for Compressive ImagingDevices”, invented by Kevin F. Kelly, Richard G. Baraniuk, LenoreMcMackin, Robert F. Bridge, Sujoy Chatterjee and Tyler H. Weston;

Terminology

A memory medium is a non-transitory medium configured for the storageand retrieval of information. Examples of memory media include: variouskinds of semiconductor-based memory such as RAM and ROM; various kindsof magnetic media such as magnetic disk, tape, strip and film; variouskinds of optical media such as CD-ROM and DVD-ROM; various media basedon the storage of electrical charge and/or any of a wide variety ofother physical quantities; media fabricated using various lithographictechniques; etc. The term “memory medium” includes within its scope ofmeaning the possibility that a given memory medium might be a union oftwo or more memory media that reside at different locations, e.g., ondifferent chips in a system or on different computers in a network.

A computer-readable memory medium may be configured so that it storesprogram instructions and/or data, where the program instructions, ifexecuted by a computer system, cause the computer system to perform amethod, e.g., any of a method embodiments described herein, or, anycombination of the method embodiments described herein, or, any subsetof any of the method embodiments described herein, or, any combinationof such subsets.

A computer system is any device (or combination of devices) having atleast one processor that is configured to execute program instructionsstored on a memory medium. Examples of computer systems include personalcomputers (PCs), workstations, laptop computers, tablet computers,mainframe computers, server computers, client computers, network orInternet appliances, hand-held devices, mobile devices, personal digitalassistants (PDAs), tablet computers, computer-based television systems,grid computing systems, wearable computers, computers implanted inliving organisms, computers embedded in head-mounted displays, computersembedded in sensors forming a distributed network, etc.

A programmable hardware element (PHE) is a hardware device that includesmultiple programmable function blocks connected via a system ofprogrammable interconnects. Examples of PHEs include FPGAs (FieldProgrammable Gate Arrays), PLDs (Programmable Logic Devices), FPOAs(Field Programmable Object Arrays), and CPLDs (Complex PLDs). Theprogrammable function blocks may range from fine grained (combinatoriallogic or look up tables) to coarse grained (arithmetic logic units orprocessor cores).

As used herein, the term “light” is meant to encompass within its scopeof meaning any electromagnetic radiation whose spectrum lies within thewavelength range [λ_(L), λ_(U)], where the wavelength range includes thevisible spectrum, the ultra-violet (UV) spectrum, infrared (IR) spectrumand the terahertz (THz) spectrum. Thus, for example, visible radiation,or UV radiation, or IR radiation, or THz radiation, or any combinationthereof is “light” as used herein.

In some embodiments, a computer system may be configured to include aprocessor (or a set of processors) and a memory medium, where the memorymedium stores program instructions, where the processor is configured toread and execute the program instructions stored in the memory medium,where the program instructions are executable by the processor toimplement a method, e.g., any of the various method embodimentsdescribed herein, or, any combination of the method embodimentsdescribed herein, or, any subset of any of the method embodimentsdescribed herein, or, any combination of such subsets.

System 100 for Operating on Light

A system 100 for operating on light may be configured as shown in FIG.2A. The system 100 may include a light modulation unit 110, a lightsensing device 130 and an analog-to-digital converter (ADC) 140.

The light modulation unit 110 is configured to modulate a receivedstream of light L with a series of spatial patterns in order to producea modulated light stream (MLS). The spatial patterns of the series maybe applied sequentially to the light stream so that successive timeslices of the light stream are modulated, respectively, with successiveones of the spatial patterns. (The action of sequentially modulating thelight stream L with the spatial patterns imposes the structure of timeslices on the light stream.)

The light modulation unit 110 includes a plurality of light modulatingelements configured to modulate corresponding portions of the lightstream. Each of the spatial patterns specifies an amount (or extent orvalue) of modulation for each of the light modulating elements.Mathematically, one might think of the light modulation unit's action ofapplying a given spatial pattern as performing an element-wisemultiplication of a light field vector (x_(ij)) representing a timeslice of the light stream L by a vector of scalar modulation values(m_(ij)) to obtain a time slice of the modulated light stream:(m_(ij))*(x_(ij))=(m_(ij)x_(ij)). The vector (m_(ij)) is specified bythe spatial pattern. Each light modulating element effectively scales(multiplies) the intensity of its corresponding light stream portion bythe corresponding scalar factor.

The light modulation unit 110 may be realized in various ways. In someembodiments, the LMU 110 may be realized by a plurality of mirrors(e.g., micromirrors) whose orientations are independently controllable.In another set of embodiments, the LMU 110 may be realized by an arrayof elements whose transmittances are independently controllable, e.g.,as with an array of LCD shutters. An electrical control signal suppliedto each element controls the extent to which light is able to transmitthrough the element. In yet another set of embodiments, the LMU 110 maybe realized by an array of independently-controllable mechanicalshutters (e.g., micromechanical shutters) that cover an array ofapertures, with the shutters opening and closing in response toelectrical control signals, thereby controlling the flow of lightthrough the corresponding apertures. In yet another set of embodiments,the LMU 110 may be realized by a perforated mechanical plate, with theentire plate moving in response to electrical control signals, therebycontrolling the flow of light through the corresponding perforations. Inyet another set of embodiments, the LMU 110 may be realized by an arrayof transceiver elements, where each element receives and thenimmediately retransmits light in a controllable fashion. In yet anotherset of embodiments, the LMU 110 may be realized by a grating light valve(GLV) device. In yet another embodiment, the LMU 110 may be realized bya liquid-crystal-on-silicon (LCOS) device.

In some embodiments, the light modulating elements are arranged in anarray, e.g., a two-dimensional array or a one-dimensional array. Any ofvarious array geometries are contemplated. For example, in someembodiments, the array is a square array or rectangular array. Inanother embodiment, the array is hexagonal. In some embodiments, thelight modulating elements are arranged in a spatially random fashion.

Let N denote the number of light modulating elements in the lightmodulation unit 110. In various embodiments, the number N may take awide variety of values. For example, in different sets of embodiments, Nmay be, respectively, in the range [64, 256], in the range [256, 1024],in the range [1024,4096], in the range [2¹²,2¹⁴], in the range[2¹⁴,2¹⁶], in the range [2¹⁶,2¹⁸], in the range [2¹⁸,2²⁰], in the range[2²⁰,2²²], in the range [2²²,2²⁴], in the range [2²⁴,2²⁶], in the rangefrom 2²⁶ to infinity. The particular value used in any given embodimentmay depend on one or more factors specific to the embodiment.

The light sensing device 130 may be configured to receive the modulatedlight stream MLS and to generate an analog electrical signal I_(MLS)(t)representing intensity of the modulated light stream as a function oftime.

The light sensing device 130 may include one or more light sensingelements. The term “light sensing element” may be interpreted as meaning“a transducer between a light signal and an electrical signal”. Forexample, a photodiode is a light sensing element. In various otherembodiments, light sensing elements might include devices such asmetal-semiconductor-metal (MSM) photodetectors, phototransistors,phototubes and photomultiplier tubes.

In some embodiments, the light sensing device 130 includes one or moreamplifiers (e.g., transimpedance amplifiers) to amplify the analogelectrical signals generated by the one or more light sensing elements.

The ADC 140 acquires a sequence of samples {I_(MLS)(k)} of the analogelectrical signal I_(MLS)(t). Each of the samples may be interpreted asan inner product between a corresponding time slice of the light streamL and a corresponding one of the spatial patterns. The set of samples{I_(MLS)(k)} comprises an encoded representation, e.g., a compressedrepresentation, of an image and may be used to reconstruct the imagebased on any reconstruction algorithm known in the field of compressivesensing. (The image is said to be “reconstructed” because it isrecognized as having previously existed, although only transiently, inthe incident light stream. By use of the term “reconstruct”, we do notmean to suggest that the image has existed in stored digital form priorto the acquisition of the samples.) To reconstruct a sequence of images,the samples of the sequence {I_(MLS)(k)} may be partitioned intocontiguous subsets, and then the subsets may be processed to reconstructcorresponding images.

In some embodiments, the samples {I_(MLS)(k)} may be used for somepurpose other than, or in addition to, image (or image sequence)reconstruction. For example, system 100 (or some other system) mayoperate on the samples to perform an inference task, such as detectingthe presence of a signal or object, identifying a signal or an object,classifying a signal or an object, estimating one or more parametersrelating to a signal or an object, tracking a signal or an object, etc.In some embodiments, an object under observation by system 100 may beidentified or classified by virtue of its sample set {I_(MLS)(k)} (orparameters derived from that sample set) being similar to one of acollection of stored sample sets (or parameter sets).

In some embodiments, the light sensing device 130 includes exactly onelight sensing element. (For example, the single light sensing elementmay be a photodiode.) The light sensing element may couple to anamplifier (e.g., a TIA) (e.g., a multi-stage amplifier).

In some embodiments, the light sensing device 130 may include aplurality of light sensing elements (e.g., photodiodes). Each lightsensing element may convert light impinging on its light sensing surfaceinto a corresponding analog electrical signal representing intensity ofthe impinging light as a function of time. In some embodiments, eachlight sensing element may couple to a corresponding amplifier so thatthe analog electrical signal produced by the light sensing element canbe amplified prior to digitization. System 100 may be configured so thateach light sensing element receives, e.g., a corresponding spatialportion (or spectral portion) of the modulated light stream.

In one embodiment, the analog electrical signals produced, respectively,by the light sensing elements may be summed to obtain a sum signal. Thesum signal may then be digitized by the ADC 140 to obtain the sequenceof samples {I_(MLS)(k)}. In another embodiment, the analog electricalsignals may be individually digitized, each with its own ADC, to obtaincorresponding sample sequences. The sample sequences may then be addedto obtain the sequence {I_(MLS)(k)}. In another embodiment, the analogelectrical signals produced by the light sensing elements may be sampledby a smaller number of ADCs than light sensing elements through the useof time multiplexing. For example, in one embodiment, system 100 may beconfigured to sample two or more of the analog electrical signals byswitching the input of an ADC among the outputs of the two or morecorresponding light sensing elements at a sufficiently high rate.

In some embodiments, the light sensing device 130 may include an arrayof light sensing elements. Arrays of any of a wide variety of sizes,configurations and material technologies are contemplated. In oneembodiment, the light sensing device 130 includes a focal plane arraycoupled to a readout integrated circuit. In one embodiment, the lightsensing device 130 may include an array of cells, where each cellincludes a corresponding light sensing element and is configured tointegrate and hold photo-induced charge created by the light sensingelement, and to convert the integrated charge into a corresponding cellvoltage. The light sensing device may also include (or couple to)circuitry configured to sample the cell voltages using one or more ADCs.

In some embodiments, the light sensing device 130 may include aplurality (or array) of light sensing elements, where each light sensingelement is configured to receive a corresponding spatial portion of themodulated light stream, and each spatial portion of the modulated lightstream comes from a corresponding sub-region of the array of lightmodulating elements. (For example, the light sensing device 130 mayinclude a quadrant photodiode, where each quadrant of the photodiode isconfigured to receive modulated light from a corresponding quadrant ofthe array of light modulating elements. As another example, the lightsensing device 130 may include a bi-cell photodiode. As yet anotherexample, the light sensing device 130 may include a focal plane array.)Each light sensing element generates a corresponding signal representingintensity of the corresponding spatial portion as a function of time.Each signal may be digitized (e.g., by a corresponding ADC, or perhapsby a shared ADC) to obtain a corresponding sequence of samples. Thus, aplurality of sample sequences are obtained, one sample sequence perlight sensing element. Each sample sequence may be processed toreconstruct a corresponding sub-image. The sub-images may be joinedtogether to form a whole image. The sample sequences may be captured inresponse to the modulation of the incident light stream with a sequenceof M spatial patterns, e.g., as variously described above. By employingany of various reconstruction algorithms known in the field ofcompressive sensing, the number of pixels in each reconstructed imagemay be greater than (e.g., much greater than) M. To reconstruct eachsub-image, the reconstruction algorithm uses the corresponding samplesequence and the restriction of the spatial patterns to thecorresponding sub-region of the array of light modulating elements.

In some embodiments, the light sensing device 130 includes a smallnumber of light sensing elements (e.g., in respective embodiments, one,two, less than 8, less than 16, less the 32, less than 64, less than128, less than 256). Because the light sensing device of theseembodiments includes a small number of light sensing elements (e.g., farless than the typical modern CCD-based or CMOS-based camera), an entityinterested in producing any of these embodiments may afford to spendmore per light sensing element to realize features that are beyond thecapabilities of modern array-based image sensors of large pixel count,e.g., features such as higher sensitivity, extended range ofsensitivity, new range(s) of sensitivity, extended dynamic range, higherbandwidth/lower response time. Furthermore, because the light sensingdevice includes a small number of light sensing elements, an entityinterested in producing any of these embodiments may use newer lightsensing technologies (e.g., based on new materials or combinations ofmaterials) that are not yet mature enough to be manufactured into focalplane arrays (FPA) with large pixel count. For example, new detectormaterials such as super-lattices, quantum dots, carbon nanotubes andgraphene can significantly enhance the performance of IR detectors byreducing detector noise, increasing sensitivity, and/or decreasingdetector cooling requirements.

In one embodiment, the light sensing device 130 is a thermo-electricallycooled InGaAs detector. (InGaAs stands for “Indium Gallium Arsenide”.)In other embodiments, the InGaAs detector may be cooled by othermechanisms (e.g., liquid nitrogen or a Sterling engine). In yet otherembodiments, the InGaAs detector may operate without cooling. In yetother embodiments, different detector materials may be used, e.g.,materials such as MCT (mercury-cadmium-telluride), InSb (IndiumAntimonide) and VOx (Vanadium Oxide).

In different embodiments, the light sensing device 130 may be sensitiveto light at different wavelengths or wavelength ranges. In someembodiments, the light sensing device 130 may be sensitive to light overa broad range of wavelengths, e.g., over the entire visible spectrum orover the entire range [λ_(L),λ_(u)] as defined above.

In some embodiments, the light sensing device 130 may include one ormore dual-sandwich photodetectors. A dual sandwich photodetectorincludes two photodiodes stacked (or layered) one on top of the other.

In one embodiment, the light sensing device 130 may include one or moreavalanche photodiodes.

In one embodiment, the light sensing device 130 may include one or morephotomultiplier tubes (PMTs).

In some embodiments, a filter may be placed in front of the lightsensing device 130 to restrict the modulated light stream to a specificrange of wavelengths or specific polarization. Thus, the signalI_(MLS)(t) generated by the light sensing device 130 may berepresentative of the intensity of the restricted light stream. Forexample, by using a filter that passes only IR ligh_(t), the lightsensing device may be effectively converted into an IR detector. Thesample principle may be applied to effectively convert the light sensingdevice into a detector for red or blue or green or UV or any desiredwavelength band, or, a detector for light of a certain polarization.

In some embodiments, system 100 includes a color wheel whose rotation issynchronized with the application of the spatial patterns to the lightmodulation unit. As it rotates, the color wheel cyclically applies anumber of optical bandpass filters to the modulated light stream MLS.Each bandpass filter restricts the modulated light stream to acorresponding sub-band of wavelengths. Thus, the samples captured by theADC 140 will include samples of intensity in each of the sub-bands. Thesamples may be de-multiplexed to form separate sub-band sequences. Eachsub-band sequence may be processed to generate a corresponding sub-bandimage. (As an example, the color wheel may include a red-pass filter, agreen-pass filter and a blue-pass filter to support color imaging.)

In some embodiments, the system 100 may include a memory (or a set ofmemories of one or more kinds).

In some embodiments, system 100 may include a processing unit 150, e.g.,as shown in FIG. 2B. The processing unit 150 may be a digital circuit ora combination of digital circuits. For example, the processing unit maybe a microprocessor (or system of interconnected of microprocessors), aprogrammable hardware element such as a field-programmable gate array(FPGA), an application specific integrated circuit (ASIC), or anycombination such elements. The processing unit 150 may be configured toperform one or more functions such as image (or image sequence)reconstruction, system control, user interface, statistical analysis,and one or more inferences tasks.

The system 100 (e.g., the processing unit 150) may store the samples{I_(MLS)(k)} in a memory, e.g., a memory resident in the system 100 orin some other system.

In one embodiment, processing unit 150 is configured to operate on thesamples {I_(MLS)(k)} to generate the image (or image sequence). In thisembodiment, the processing unit 150 may include a microprocessorconfigured to execute software (i.e., program instructions), especiallysoftware for executing an image reconstruction algorithm. In oneembodiment, system 100 is configured to transmit the compensated samplesto some other system through a communication channel. (In embodimentswhere the spatial patterns are randomly-generated, system 100 may alsotransmit the random seed(s) used to generate the spatial patterns.) Thatother system may operate on the samples to reconstruct the image (orimage sequence). System 100 may have one or more interfaces configuredfor sending (and perhaps also receiving) data through one or morecommunication channels, e.g., channels such as wireless channels, wiredchannels, fiber optic channels, acoustic channels, laser-based channels,etc.

In some embodiments, processing unit 150 is configured to use any of avariety of algorithms and/or any of a variety of transformations toperform image (or image sequence) reconstruction. System 100 may allow auser to choose a desired algorithm and/or a desired transformation forperforming the image (or image sequence) reconstruction.

In some embodiments, the system 100 is configured to acquire a set Z_(M)of samples from the ADC 140 so that the sample set Z_(M) corresponds toM of the spatial patterns applied to the light modulation unit 110,where M is a positive integer. The number M is selected so that thesample set Z_(M) is useable to reconstruct an n-pixel image or n-pixelimage sequence that represents the incident light stream, where n is apositive integer less than or equal to the number N of light modulatingelements in the light modulation unit 110. System 100 may be configuredso that the number M is smaller than n. Thus, system 100 may operate asa compressive sensing device. (The number of “pixels” in an imagesequence is the number of images in the image sequence times the numberof pixels per image, or equivalently, the sum of the pixel counts of theimages in the image sequence.)

In various embodiments, the compression ratio M/n may take any of a widevariety of values. For example, in different sets of embodiments, M/nmay be, respectively, in the range [0.9,0.8], in the range [0.8,0.7], inthe range [0.7,0.6], in the range [0.6,0.5], in the range [0.5,0.4], inthe range [0.4,0.3], in the range [0.3,0.2], in the range [0.2,0.1], inthe range [0.1,0.05], in the range [0.05,0.01], in the range[0.001,0.01].

As noted above, the image reconstructed from the sample subset Z_(M) maybe an n-pixel image with n≦N. The spatial patterns may be designed tosupport a value of n less than N, e.g., by forcing the array of lightmodulating elements to operate at a lower effective resolution than thephysical resolution N. For example, the spatial patterns may be designedto force each 2×2 cell of light modulating elements to act in unison. Atany given time, the modulation state of the four elements in a 2×2 cellwill agree. Thus, the effective resolution of the array of lightmodulating elements is reduced to N/4. This principle generalizes to anycell size, to cells of any shape, and to collections of cells withnon-uniform cell size and/or cell shape. For example, a collection ofcells of size k_(H)xk_(v), where k_(H) and k_(v) are positive integers,would give an effective resolution equal to N/(k_(H)k_(v)). In onealternative embodiment, cells near the center of the array may havesmaller sizes than cells near the periphery of the array.

Another way the spatial patterns may be arranged to support thereconstruction of an n-pixel image with n less than N is to allow thespatial patterns to vary only within a subset of the array of lightmodulating elements. In this mode of operation, the spatial patterns arenull (take the value zero) outside the subset. (Control unit 120 may beconfigured to implement this restriction of the spatial patterns.) Lightmodulating elements corresponding to positions outside of the subset donot send any light (or send only the minimum amount of light attainable)to the light sensing device. Thus, the reconstructed image is restrictedto the subset. In some embodiments, each spatial pattern (e.g., of ameasurement pattern sequence) may be multiplied element-wise by a binarymask that takes the one value only in the allowed subset, and theresulting product pattern may be supplied to the light modulation unit.In some embodiments, the subset is a contiguous region of the array oflight modulating elements, e.g., a rectangle or a circular disk or ahexagon. In some embodiments, the size and/or position of the region mayvary (e.g., dynamically). The position of the region may vary in orderto track a moving object. The size of the region may vary in order todynamically control the rate of image acquisition or frame rate. In someembodiments, the size of the region may be determined by user input. Forexample, system 100 may provide an input interface (GUI and/ormechanical control device) through which the user may vary the size ofthe region over a continuous range of values (or alternatively, adiscrete set of values), thereby implementing a digital zoom function.Furthermore, in some embodiments, the position of the region within thefield of view may be controlled by user input.

In one embodiment, system 100 may include a light transmitter configuredto generate a light beam (e.g., a laser beam), to modulate the lightbeam with a data signal and to transmit the modulated light beam intospace or onto an optical fiber. System 100 may also include a lightreceiver configured to receive a modulated light beam from space or froman optical fiber, and to recover a data stream from the receivedmodulated light beam.

In one embodiment, system 100 may be configured as a low-cost sensorsystem having minimal processing resources, e.g., processing resourcesinsufficient to perform image (or image sequence) reconstruction inuser-acceptable time. In this embodiment, the system 100 may storeand/or transmit the samples {I_(MLS)(k)} so that another agent, moreplentifully endowed with processing resources, may perform the image (orimage sequence) reconstruction based on the samples.

In some embodiments, system 100 may include an optical subsystem 105that is configured to modify or condition the light stream L before itarrives at the light modulation unit 110, e.g., as shown in FIG. 2C. Forexample, the optical subsystem 105 may be configured to receive thelight stream L from the environment and to focus the light stream onto amodulating plane of the light modulation unit 110. The optical subsystem105 may include a camera lens (or a set of lenses). The lens (or set oflenses) may be adjustable to accommodate a range of distances toexternal objects being imaged/sensed/captured. The optical subsystem 105may allow manual and/or digital control of one or more parameters suchas focus, zoom, shutter speed and f-stop.

In some embodiments, system 100 may include an optical subsystem 117 todirect the modulated light stream MLS onto a light sensing surface (orsurfaces) of the light sensing device 130.

In some embodiments, the optical subsystem 117 may include one or morelenses, and/or, one or more mirrors.

In some embodiments, the optical subsystem 117 is configured to focusthe modulated light stream onto the light sensing surface (or surfaces).The term “focus” implies an attempt to achieve the condition that rays(photons) diverging from a point on an object plane converge to a point(or an acceptably small spot) on an image plane. The term “focus” alsotypically implies continuity between the object plane point and theimage plane point (or image plane spot); points close together on theobject plane map respectively to points (or spots) close together on theimage plane. In at least some of the system embodiments that include anarray of light sensing elements, it may be desirable for the modulatedlight stream MLS to be focused onto the light sensing array so thatthere is continuity between points on the light modulation unit LMU andpoints (or spots) on the light sensing array.

In some embodiments, the optical subsystem 117 may be configured todirect the modulated light stream MLS onto the light sensing surface (orsurfaces) of the light sensing device 130 in a non-focusing fashion. Forexample, in a system embodiment that includes only one photodiode, itmay not be so important to achieve the “in focus” condition at the lightsensing surface of the photodiode since positional information ofphotons arriving at that light sensing surface will be immediately lost.

In one embodiment, the optical subsystem 117 may be configured toreceive the modulated light stream and to concentrate the modulatedlight stream into an area (e.g., a small area) on a light sensingsurface of the light sensing device 130. Thus, the diameter of themodulated light stream may be reduced (possibly, radically reduced) inits transit from the optical subsystem 117 to the light sensing surface(or surfaces) of the light sensing device 130. For example, in someembodiments, the diameter may be reduced by a factor of more than 1.5to 1. In other embodiments, the diameter may be reduced by a factor ofmore than 2 to 1. In yet other embodiments, the diameter may be reducedby a factor of more than 10 to 1. In yet other embodiments, the diametermay be reduced by factor of more than 100 to 1. In yet otherembodiments, the diameter may be reduced by factor of more than 400to 1. In one embodiment, the diameter is reduced so that the modulatedlight stream is concentrated onto the light sensing surface of a singlelight sensing element (e.g., a single photodiode).

In some embodiments, this feature of concentrating the modulated lightstream onto the light sensing surface (or surfaces) of the light sensingdevice allows the light sensing device to sense at any given time thesum (or surface integral) of the intensities of the modulated lightportions within the modulated light stream. (Each time slice of themodulated light stream comprises a spatial ensemble of modulated lightportions due to the modulation unit's action of applying thecorresponding spatial pattern to the light stream.)

In some embodiments, the modulated light stream MLS may be directed ontothe light sensing surface of the light sensing device 130 withoutconcentration, i.e., without decrease in diameter of the modulated lightstream, e.g., by use of photodiode having a large light sensing surface,large enough to contain the cross section of the modulated light streamwithout the modulated light stream being concentrated.

In some embodiments, the optical subsystem 117 may include one or morelenses. FIG. 2E shows an embodiment where optical subsystem 117 isrealized by a lens 117L, e.g., a biconvex lens or a condenser lens.

In some embodiments, the optical subsystem 117 may include one or moremirrors. In one embodiment, the optical subsystem 117 includes aparabolic mirror (or spherical mirror) to concentrate the modulatedlight stream onto a neighborhood (e.g., a small neighborhood) of theparabolic focal point. In this embodiment, the light sensing surface ofthe light sensing device may be positioned at the focal point.

In some embodiments, system 100 may include an optical mechanism (e.g.,an optical mechanism including one or more prisms and/or one or morediffraction gratings) for splitting or separating the modulated lightstream MLS into two or more separate streams (perhaps numerous streams),where each of the streams is confined to a different wavelength range.The separate streams may each be sensed by a separate light sensingdevice. (In some embodiments, the number of wavelength ranges may be,e.g., greater than 8, or greater than 16, or greater than 64, or greaterthan 256, or greater than 1024.) Furthermore, each separate stream maybe directed (e.g., focused or concentrated) onto the corresponding lightsensing device as described above in connection with optical subsystem117. The samples captured from each light sensing device may be used toreconstruct a corresponding image (or image sequence) for thecorresponding wavelength range. In one embodiment, the modulated lightstream is separated into red, green and blue streams to support color(R,G,B) measurements. In another embodiment, the modulated light streammay be separated into IR, red, green, blue and UV streams to supportfive-channel multi-spectral imaging: (IR, R, G, B, UV). In someembodiments, the modulated light stream may be separated into a numberof sub-bands (e.g., adjacent sub-bands) within the IR band to supportmulti-spectral or hyper-spectral IR imaging. In some embodiments, thenumber of IR sub-bands may be, e.g., greater than 8, or greater than 16,or greater than 64, or greater than 256, or greater than 1024. In someembodiments, the modulated light stream may experience two or morestages of spectral separation. For example, in a first stage themodulated light stream may be separated into an IR stream confined tothe IR band and one or more additional streams confined to other bands.In a second stage, the IR stream may be separated into a number ofsub-bands (e.g., numerous sub-bands) (e.g., adjacent sub-bands) withinthe IR band to support multispectral or hyper-spectral IR imaging.

In some embodiments, system 100 may include an optical mechanism (e.g.,a mechanism including one or more beam splitters) for splitting orseparating the modulated light stream MLS into two or more separatestreams, e.g., where each of the streams have the same (or approximatelythe same) spectral characteristics or wavelength range. The separatestreams may then pass through respective bandpass filters to obtaincorresponding modified streams, where each modified stream is restrictedto a corresponding band of wavelengths. Each of the modified streams maybe sensed by a separate light sensing device. (In some embodiments, thenumber of wavelength bands may be, e.g., greater than 8, or greater than16, or greater than 64, or greater than 256, or greater than 1024.)Furthermore, each of the modified streams may be directed (e.g., focusedor concentrated) onto the corresponding light sensing device asdescribed above in connection with optical subsystem 117. The samplescaptured from each light sensing device may be used to reconstruct acorresponding image (or image sequence) for the corresponding wavelengthband. In one embodiment, the modulated light stream is separated intothree streams which are then filtered, respectively, with a red-passfilter, a green-pass filter and a blue-pass filter. The resulting red,green and blue streams are then respectively detected by three lightsensing devices to support color (R,G,B) acquisition. In another similarembodiment, five streams are generated, filtered with five respectivefilters, and then measured with five respective light sensing devices tosupport (IR, R, G, B, UV) multi-spectral acquisition. In yet anotherembodiment, the modulated light stream of a given band may be separatedinto a number of (e.g., numerous) sub-bands to support multi-spectral orhyper-spectral imaging.

In some embodiments, system 100 may include an optical mechanism forsplitting or separating the modulated light stream MLS into two or moreseparate streams. The separate streams may be directed to (e.g.,concentrated onto) respective light sensing devices. The light sensingdevices may be configured to be sensitive in different wavelengthranges, e.g., by virtue of their different material properties. Samplescaptured from each light sensing device may be used to reconstruct acorresponding image (or image sequence) for the corresponding wavelengthrange.

In some embodiments, system 100 may include a control unit 120configured to supply the spatial patterns to the light modulation unit110, as shown in FIG. 2F. The control unit may itself generate thepatterns or may receive the patterns from some other agent. The controlunit 120 and the ADC 140 may be controlled by a common clock signal sothat ADC 140 can coordinate (synchronize) its action of capturing thesamples {I_(MLS)(k)} with the control unit's action of supplying spatialpatterns to the light modulation unit 110. (System 100 may include clockgeneration circuitry.)

In some embodiments, the control unit 120 may supply the spatialpatterns to the light modulation unit in a periodic fashion.

The control unit 120 may be a digital circuit or a combination ofdigital circuits. For example, the control unit may include amicroprocessor (or system of interconnected of microprocessors), aprogrammable hardware element such as a field-programmable gate array(FPGA), an application specific integrated circuit (ASIC), or anycombination such elements.

In some embodiments, the control unit 120 may include a random numbergenerator (RNG) or a set of random number generators to generate thespatial patterns or some subset of the spatial patterns.

In some embodiments, system 100 is battery powered. In some embodiments,the system 100 includes a set of one or more solar cells and associatedcircuitry to derive power from sunlight.

In some embodiments, system 100 includes its own light source forilluminating the environment or a target portion of the environment.

In some embodiments, system 100 may include a display (or an interfaceconfigured for coupling to a display) for displaying reconstructedimages or image sequences, e.g., video sequences.

In some embodiments, system 100 may include one or more input devices(and/or, one or more interfaces for input devices), e.g., anycombination or subset of the following devices: a set of buttons and/orknobs, a keyboard, a keypad, a mouse, a touch-sensitive pad such as atrackpad, a touch-sensitive display screen, one or more microphones, oneor more temperature sensors, one or more chemical sensors, one or morepressure sensors, one or more accelerometers, one or more orientationsensors (e.g., a three-axis gyroscopic sensor), one or more proximitysensors, one or more antennas, etc.

Regarding the spatial patterns that are used to modulate the lightstream L, it should be understood that there are a wide variety ofpossibilities. In some embodiments, the control unit 120 may beprogrammable so that any desired set of spatial patterns may be used.

In some embodiments, the spatial patterns are binary valued. Such anembodiment may be used, e.g., when the light modulating elements aretwo-state devices. In some embodiments, the spatial patterns are n-statevalued, where each element of each pattern takes one of n states, wheren is an integer greater than two. (Such an embodiment may be used, e.g.,when the light modulating elements are each able to achieve n or moremodulation states). In some embodiments, the spatial patterns are realvalued, e.g., when each of the light modulating elements admits acontinuous range of modulation. (It is noted that even a two-statemodulating element may be made to effectively apply a continuous rangeof modulation by duty cycling the two states during modulationintervals.)

The spatial patterns may belong to a set of measurement vectors that isincoherent with a set of vectors in which the image is approximatelysparse (“the sparsity vector set”). (See “Sparse Signal Detection fromIncoherent Projections”, Proc. Int. Conf. Acoustics, Speech SignalProcessing—ICASSP, May 2006, Duarte et al.) Given two sets of vectorsA={a_(i)} and B={b_(i)} in the same N-dimensional space, A and B aresaid to be incoherent if their coherence measure μ(A,B) is sufficientlysmall. Assuming that the vectors {a_(i)} and the vectors {b_(i)} haveunit L² norm, the coherence measure is defined as:

${\mu\left( {A,B} \right)} = {\max\limits_{i,j}{{\left\langle {a_{i},b_{j}} \right\rangle }.}}$

The number of compressive sensing measurements (i.e., samples of thesequence {I_(MLS)(k)} needed to reconstruct an N-pixel image thataccurately represents the scene being captured is a strictly increasingfunction of the coherence between the measurement vector set and thesparsity vector set. Thus, better compression can be achieved withsmaller values of the coherence. (The measurement vector set may also bereferred to herein as the “measurement pattern set”. Likewise, thesparsity vector set may also be referred to herein as the “sparsitypattern set”.)

In some embodiments, the measurement vector set may be based on a code.Any of various codes from information theory may be used, e.g., codessuch as exponentiated Kerdock codes, exponentiated Delsarte-Goethalscodes, run-length limited codes, LDPC codes, Reed Solomon codes and ReedMuller codes.

In some embodiments, the measurement vector set corresponds to arandomized or permuted basis, where the basis may be, for example, theDiscrete Cosine Transform (DCT) basis or Hadamard basis.

In some embodiments, the spatial patterns may be random or pseudo-randompatterns, e.g., generated according to a random number generation (RNG)algorithm using one or more seeds. In some embodiments, the elements ofeach pattern are generated by a series of Bernoulli trials, where eachtrial has a probability p of giving the value one and probability 1-p ofgiving the value zero. (For example, in one embodiment p=½.) In someembodiments, the elements of each pattern are generated by a series ofdraws from a Gaussian random variable.)

The system 100 may be configured to operate in a compressive fashion,where the number of the samples {I_(MLS)(k)} captured by the system 100is less than (e.g., much less than) the number of pixels in the image(or image sequence) to be reconstructed from the samples. In manyapplications, this compressive realization is very desirable because itsaves on power consumption, memory utilization and transmissionbandwidth consumption. However, non-compressive realizations arecontemplated as well.

In some embodiments, the system 100 is configured as a camera or imagerthat captures information representing an image (or a series of images)from the external environment, e.g., an image (or a series of images) ofsome external object or scene. The camera system may take differentforms in different application domains, e.g., domains such as visiblelight photography, infrared photography, ultraviolet photography,high-speed photography, low-light photography, underwater photography,multi-spectral imaging, hyper-spectral imaging, etc. In someembodiments, system 100 is configured to operate in conjunction with (oras part of) another system, e.g., in conjunction with (or as part of) amicroscope, a telescope, a robot, a security system, a surveillancesystem, a fire sensor, a node in a distributed sensor network, etc.

In some embodiments, system 100 is configured as a spectrometer.

In some embodiments, system 100 is configured as a multi-spectral orhyper-spectral imager.

In some embodiments, system 100 may also be configured to operate as aprojector. Thus, system 100 may include a light source, e.g., a lightsource located at or near a focal point of optical subsystem 117. Inprojection mode, the light modulation unit 110 may be supplied with animage (or a sequence of images), e.g., by control unit 120. The lightmodulation unit may receive a light beam generated by the light source,and modulate the light beam with the image (or sequence of images) toobtain a modulated light beam. The modulated light beam exits the system100 and is displayed on a display surface (e.g., an external screen).

In one embodiment, the light modulation unit 110 may receive the lightbeam from the light source and modulate the light beam with a timesequence of spatial patterns (from a measurement pattern set). Theresulting modulated light beam exits the system 100 and is used toilluminate the external scene. Light reflected from the external scenein response to the modulated light beam is measured by a light sensingdevice (e.g., a photodiode). The samples captured by the light sensingdevice comprise compressive measurements of external scene. Thosecompressive measurements may be used to reconstruct an image or imagesequence as variously described above.

In some embodiments, system 100 includes an interface for communicatingwith a host computer. The host computer may send control informationand/or program code to the system 100 via the interface. Furthermore,the host computer may receive status information and/or compressivesensing measurements from system 100 via the interface.

In one realization 200 of system 100, the light modulation unit 110 maybe realized by a plurality of mirrors, e.g., as shown in FIG. 3A. (Themirrors are collectively indicated by the label 110M.) The mirrors 110Mare configured to receive corresponding portions of the light L receivedfrom the environment, albeit not necessarily directly from theenvironment. (There may be one or more optical elements, e.g., one ormore lenses along the input path to the mirrors 110M.) Each of themirrors is configured to controllably switch between at least twoorientation states. In addition, each of the mirrors is configured to(a) reflect the corresponding portion of the light onto a sensing path115 when the mirror is in a first of the two orientation states and (b)reflect the corresponding portion of the light away from the sensingpath when the mirror is in a second of the two orientation states.

In some embodiments, the mirrors 110M are arranged in an array, e.g., atwo-dimensional array or a one-dimensional array. Any of various arraygeometries are contemplated. For example, in different embodiments, thearray may be a square array, a rectangular array, a hexagonal array,etc. In some embodiments, the mirrors are arranged in a spatially-randomfashion.

The mirrors 110M may be part of a digital micromirror device (DMD). Forexample, in some embodiments, one of the DMDs manufactured by TexasInstruments may be used.

The control unit 120 may be configured to drive the orientation statesof the mirrors through the series of spatial patterns, where each of thepatterns of the series specifies an orientation state for each of themirrors.

The light sensing device 130 may be configured to receive the lightportions reflected at any given time onto the sensing path 115 by thesubset of mirrors in the first orientation state and to generate ananalog electrical signal I_(MLS)(t) representing a cumulative intensityof the received light portions as function of time. As the mirrors aredriven through the series of spatial patterns, the subset of mirrors inthe first orientation state will vary from one spatial pattern to thenext. Thus, the cumulative intensity of light portions reflected ontothe sensing path 115 and arriving at the light sensing device will varyas a function time. Note that the term “cumulative” is meant to suggesta summation (spatial integration) over the light portions arriving atthe light sensing device at any given time. This summation may beimplemented, at least in part, optically (e.g., by means of a lensand/or mirror that concentrates or focuses the light portions onto aconcentrated area as described above).

System realization 200 may include any subset of the features,embodiments and elements discussed above with respect to system 100. Forexample, system realization 200 may include the optical subsystem 105 tooperate on the incoming light L before it arrives at the mirrors 110M,e.g., as shown in FIG. 3B.

In some embodiments, system realization 200 may include the opticalsubsystem 117 along the sensing path as shown in FIG. 4. The opticalsubsystem 117 receives the light portions reflected onto the sensingpath 115 and directs (e.g., focuses or concentrates) the received lightportions onto a light sensing surface (or surfaces) of the light sensingdevice 130. In one embodiment, the optical subsystem 117 may include alens 117L, e.g., as shown in FIG. 5A.

In some embodiments, the optical subsystem 117 may include one or moremirrors, e.g., a mirror 117M as shown in FIG. 5B. Thus, the sensing pathmay be a bent path having more than one segment. FIG. 5B also shows onepossible embodiment of optical subsystem 105, as a lens 105L.

In some embodiments, there may be one or more optical elementsintervening between the optical subsystem 105 and the mirrors 110M. Forexample, as shown in FIG. 5C, a TIR prism pair 107 may be positionedbetween the optical subsystem 105 and the mirrors 110M. (TIR is anacronym for “total internal reflection”.) Light from optical subsystem105 is transmitted through the TIR prism pair and then interacts withthe mirrors 110M. After having interacted with the mirrors 110M, lightportions from mirrors in the first orientation state are reflected by asecond prism of the pair onto the sensing path 115. Light portions frommirrors in the second orientation state may be reflected away from thesensing path.

Separating Light Before Modulation to Detect Light Variation when theModulator is Powered Off

In one set of embodiments, a system 600 may be configured as illustratedin FIG. 6. System 600 may include an optical subsystem 610, a lightmodulation unit 620, a light sensing device 630, a light sensing device640 and a control unit 650. (Furthermore, system 600 may include anysubset of the features, embodiments and elements discussed above withrespect to system 100 and system realization 200 and discussed belowwith respect to system 1700.) System 600 may be configured tocompressively acquire image information from a received stream of light.However, in order to save power, the light modulation unit 620 may bemaintained in a powered-off state until the occurrence of a lightvariation event. The light variation event is indicative of thepotential presence of an object of interest in the field of view orscene under observation. Furthermore, after turning on power to thelight modulation unit, power may be saved by selectively, notcontinuously, performing image reconstruction and/or transmission of thecompressively-acquired information. For example, the data streamacquired by the light sensing device 630 may be monitored to detectobject motion and/or the occurrence of a signal of interest in the fieldof view. The reconstruction and/or transmission processes may be invokedin response to such conditions. The reconstruction algorithm may becomputationally intensive. Thus, the system may save a considerableamount of power, e.g., battery life, by running the reconstructionalgorithm selectively, not continuously. Similarly, the system may savepower by using the transmitter selectively (when needed), notcontinuously. Furthermore, the system may save power by transmittingcompressively-acquired measurements to a remote computer and invokingexecution of the reconstruction algorithm on the remote computer insteadof executing the reconstruction algorithm itself.

In some embodiments, the system 600 may be configured as a surveillancesystem or security monitoring system.

The optical subsystem 610 may be configured to receive an incident lightstream L and to separate the incident light stream into a light streamS₁ and a light stream S₂. The light stream S₁ is supplied to the lightmodulation unit 620, and the light stream S₂ is supplied to the lightsensing device 640. The separation may be performed in any of a widevariety of ways, using any of a wide variety of optical components orcombination of optical components. In some embodiments, the separationis performed so that each of the light streams S₁ and S₂ has the samefield of view into the external environment. For example, the separationmay be performed so that each ray in the incident light stream is splitinto two parts, one part entering the light stream S₁ and the other partentering the light stream S₂.

In some embodiments, the optical subsystem 610 may separate the incidentlight stream so that the light streams S₁ and S₂ are spectrally similarto the incident light stream L but of lower power than the incidentlight stream. (Two light streams are said to be spectrally similar whenthe wavelength spectrum of one is a scalar multiple of the wavelengthspectrum of the other.)

In some embodiments, the optical subsystem 610 may separate the incidentlight stream so that the light stream S₁ and/or the light stream S₂ is(are) spectrally different from incident light stream L. For example,the light stream S₂ may be restricted to the IR band while the incidentlight stream L is a broadband stream including visible light as well asIR light.

In some embodiments, the optical subsystem 610 may separate the incidentlight stream so that the light streams S₁ and S₂ are spectrallydifferent from each other. In some embodiments, the optical subsystem610 may separate the incident light stream so that the light streams S₁and S₂ are restrictions of the incident light stream L to differentwavelength bands.

In some embodiments, the optical subsystem 610 may include one or moreoptical devices such as beam splitters, prisms, TIR prisms, diffractiongratings, mirrors (e.g., partially transmitting mirrors), etc.

In some embodiments, the optical subsystem 610 may be configured in anyof the various ways described in connection with “optical subsystem1310” of U.S. patent application Ser. No. 13/193,553, filed on Jul. 28,2011, entitled “Determining Light Level Variation in Compressive Imagingby Injecting Calibration Patterns into Pattern Sequence”, which ishereby incorporated by reference in its entirety.

The light modulation unit 620 may include a plurality of lightmodulating elements (e.g., as variously described above in connectionwith light modulation unit 110). In some embodiments, the lightmodulation unit 620 includes a plurality of light reflecting elementswhose orientations are independently controllable, e.g., as variouslydescribed above in connection with mirrors 110M. The light modulationunit 620 may be, e.g., a digital micromirror device (DMD). When thelight modulation unit is powered on (i.e., is in the power-on state),each of the light reflecting elements is configured to controllablyswitch between two active orientation states. Furthermore, when it ispowered on, the light modulation unit is configured to modulate thelight stream S₁ with a time sequence of spatial patterns to obtain amodulated light stream MLS.

The light sensing device 630 may be configured to receive the modulatedlight stream MLS and to generate a data stream D₁ in response to themodulated light stream when the light modulation unit 620 is powered on.In some embodiments, the light sensing device 630 may include only onelight sensing element and an analog-to-digital converter, e.g., asvariously described above. In other embodiments, the light sensingdevice 630 may include a plurality (or an array) of light sensingelements and one or more analog-to-digital converters, e.g., asvariously described above. In some embodiments, the light sensing device630 may include a focal plane array (FPA).

One or more devices (e.g., optical devices) may intervene on the opticalpath between the light modulation unit 620 and the light sensing device630, to focus, direct or concentrate the modulated light stream MLS ontothe light-sensing surface(s) of the light sensing device 630, e.g., asvariously described above.

The light sensing device 640 may be configured to receive the lightstream S₂ and to generate a data stream D₂ in response to the lightstream S₂ at least when the light modulation unit 620 is powered off(i.e., is in the powered-off state). In some embodiments, the lightsensing device 640 may include only one light sensing element (such as aphotodiode) and an analog-to-digital converter (e.g., as variouslydescribed above). In other embodiments, the light sensing device 640includes a plurality (or array) of light sensing elements and one ormore analog-to-digital converters. For example, in one embodiment, thelight sensing device 640 may include a focal plane array (FPA).

One or more devices (e.g., optical devices) may intervene on the opticalpath between the optical subsystem 610 and the light sensing device 640to focus, direct or concentrate the light stream S₂ onto thelight-sensing surface(s) of the light sensing device 640, e.g., asvariously described above.

The control unit 650 may be configured to monitor the data stream D₂when the light modulation unit is powered off in order to detect a lightvariation event in the data stream D₂. The light variation eventindicates a variation of light in the light stream S₂, e.g., a temporalvariation having sufficient strength and/or a variation having certainspatial and/or spectral properties. (“Sufficiently strong” means, e.g.,that the magnitude of the light variation is greater than apredetermined threshold.) The control unit 650 may be further configuredto turn on power to the light modulation unit 620 in response todetermining that a trigger condition is satisfied. The trigger conditionmay include the detection of the light variation event. (In someembodiments, the trigger condition may involve one or more otherconditions in addition to the detection of the light variation event. Inother embodiments, the trigger condition is the same thing as thedetection of the light variation event.) By maintaining the lightmodulation unit in a power-off state until the occurrence of a lightvariation event, system 600 conserves power when there is likely nothingof interest occurring in the scene under observation.

The control unit 650 may be realized in any of various forms. Forexample, the control unit may include a microprocessor (or system ofinterconnected microprocessors), one or more programmable hardwareelements such as field-programmable gate arrays (FPGAs), custom-designeddigital circuitry such as one or more application specific integratedcircuits (ASICs), or any combination such elements.

In some embodiments, the control unit 650 may be configured to inject asequence of measurement patterns into the time sequence of spatialpatterns after turning on power to the light modulation unit, and toexecute a reconstruction algorithm on a subset of the data stream D₁(i.e., a subset that corresponds to the sequence of measurementpatterns) in order to obtain an image or image sequence. By saying thatthe sequence of measurement patterns is “injected” into the timesequence of spatial patterns, we mean to imply that the time sequence ofspatial patterns is free to include spatial patterns other than themeasurement patterns. For example, the time sequence of spatial patternsmay include calibration patterns and/or bright-spot search patterns inaddition to the measurement patterns. Thus, it is not necessary that thetime sequence of spatial patterns be entirely composed of measurementpatterns. The reconstructed image or image sequence represents the sceneunder observation. The measurement patterns may be configured asvariously described above. (See the above discussion of the “measurementvector set”.)

In some embodiments, the system 600 may also include a transmitter 660,e.g., as shown in FIG. 7. The control unit 650 may be configured toinject a sequence of measurement patterns into the time sequence ofspatial patterns after turning on power to the light modulation unit620, and to direct the transmitter 660 to transmit a subset of the datastream D₁ onto a transmission channel, i.e., a subset of the data streamD₁ that corresponds to the sequence of measurement patterns. (Thesamples forming the subset are acquired in response to the applicationof the sequence of measurement patterns to the light stream S₁ by thelight modulation unit 620.)

In some embodiments, the transmitter 660 may be configured to transmitthe subset of the data stream D₁ (or the entirety of the data stream D₁)to a remote system 670. The remote system may be equipped withprocessing resources (e.g., one or more processors configured to executeprogram instructions) to perform image (or image sequence)reconstruction based on the subset. Thus, system 600 may save power bynot performing the reconstruction itself.

In different embodiments, the transmitter 660 may be configured fortransmission over respectively different kinds of communication channel.For example, in some embodiments, the transmitter transmitselectromagnetic signals (e.g., radio signals or optical signals) througha wireless or wired channel. In one embodiment, the transmittertransmits electromagnetic signals through an electrical cable. Inanother embodiment, the transmitter transmits electromagnetic wavesthrough free space (e.g., the atmosphere). In yet another embodiment,the transmitter transmits through free space or through an optical fiberusing modulated light signals or modulated laser signals. In yet anotherembodiment, the transmitter transmits acoustic signals through anacoustic medium, e.g., a body of water. The transmitter may be any typeof transmitter known in the art of telecommunications.

In some embodiments, system 600 also includes a receiver as well as atransmitter to permit two-way communication with one or more otherparties.

In some embodiments, the light sensing device 640 may include a singlelight sensing element and an analog-to-digital converter (ADC), e.g., asvariously described above in connection with light sensing device 130and ADC 140. The light sensing element is configured to generate ananalog electrical signal representing intensity of the light stream S₂as a function of time. The ADC is configured to capture a sequence ofsamples of the analog electrical signal. (Each sample represents theintensity of the light stream S₂ at a corresponding time.) The datastream D₂ may include the sequence of samples. The control unit 650 maymonitor the sequence of samples (or, one or more time derivatives of thesequence of samples) to detect the light variation event. For example,the control unit may apply a low pass filter to the sequence of samples(to attenuate high-frequency noise) and then compute a time derivativeof the filtered signal. When the absolute value (or the square) of thetime derivative exceeds a detection threshold, the control unit maydeclare that the light variation event has occurred.

In some embodiments, the light variation event is interpreted as avariation in the total intensity of the light stream S₂.

In some embodiments, the light sensing device 640 may be configured inany of the various ways described in connection with “light sensingdevice 1320” of U.S. patent application Ser. No. 13/193,553, filed onJul. 28, 2011, entitled “Determining Light Level Variation inCompressive Imaging by Injecting Calibration Patterns into PatternSequence”. See especially FIGS. 13A through 17B and the correspondingtextual description in that patent application.

In some embodiments, the light sensing device 640 includes a lightsensing element and analog electrical circuitry. The light sensingelement is configured to generate an analog electrical signalrepresenting intensity of the light stream S₂ as a function of time. Theanalog electrical circuitry operates on the analog electrical signal andperforms the function of motion detection. The analog electricalcircuitry may generate a decision signal that represents at any giventime whether or not the motion is present in the field of viewrepresented by the light stream S₂.

In some embodiments, the light sensing device 640 may include aplurality (e.g., an array) of light sensing elements, each configured toreceive a corresponding spatial portion of the light stream S₂. (Forexample, the plurality of light sensing elements may be arranged tocover a cross section of the light stream S₂. Each light sensing elementthus receives the portion of the light stream S₂ that impinges upon itssurface.) For each of the light sensing elements, the light sensingdevice 640 may be configured to capture a corresponding sequence ofsamples representing intensity over time of the corresponding spatialportion of the light stream S₂. The data stream D₂ may include thesesequences of samples.

In one embodiment, the light sensing device 640 may include an M×N arrayof light sensing elements, where M is a positive integer, N is apositive integer and the product MN is greater than or equal to two.Each of the MN light sensing elements captures samples for acorresponding region of the light stream S₂. For example, in the caseM=N=2, each light sensing element may capture samples for acorresponding quadrant of the light stream S₂.

In one embodiment, the control unit 650 may detect the light variationevent by: (a) computing a weighted combination of the sequences ofintensity samples (captured from the respective light sensing elements)to obtain a composite signal, and (b) comparing the absolute value ofthe time derivative of the composite signal to a detection threshold.For example, the sample sequence(s) corresponding to one half of thefield of view may be weighted positively while the sample sequence(s)corresponding to other half may be weighted negatively. As anotherexample, the field of view may be divided into four quadrants. (In thisexample, the light sensing device 640 includes at least four lightsensing elements.) Sample sequences from the northeast and southwestquadrants may be weighted positively while sample sequences from thenorthwest and southeast quadrants may be weighted negatively.

In another embodiment, the control unit 650 may detect the lightvariation event by: computing a time derivative (or a smoothed timederivative) of each of the sample sequences to obtain a correspondingtime derivative signal, and determine if the absolute value (or square)of at least one of the time derivative signals exceeds a predeterminedthreshold.

In yet another embodiment, the data stream D₂ may include a sequence offrames, with each frame including a sample from each of the lightsensing elements of the light sensing device 640. The control unit 650may monitor the data stream D₂ by analyzing the sequence of frames. Forexample, the control unit 650 may detect the light variation event bycomputing motion vectors between successive frames of the frame sequencein a manner that is used in the MPEG video encoding algorithm. Anaverage magnitude (or a statistic) of the motion vectors may be comparedto a threshold. The light variation event may be declared when thethreshold is exceeded.

In some embodiments, the light sensing device 640 may be configured asvariously described in U.S. patent application Ser. No. 13/197,304,filed on Aug. 3, 2011, entitled “Decreasing Image Acquisition Time forCompressive Imaging Devices”, which is hereby incorporated by referencein its entirety. For example, light sensing device 640 may include the“light sensing device 630” and the “sampling subsystem 640” described inthat patent application.

In some embodiments, the light sensing device 640 may include aplurality (e.g., an array) of light sensing elements, each configured toreceive a corresponding spectral portion of the light stream S₂.Furthermore, for each of the light sensing elements, the light sensingdevice 640 may be configured to capture a corresponding sequence ofintensity samples representing intensity over time of the correspondingspectral portion of the light stream S₂. The data stream D₂ may includethese sequences of samples.

One or more optical devices may intervene along the optical path betweenthe optical subsystem 610 and the light sensing device 640 in order tospatially separate the light stream S₂ into a set or continuousdistribution of wavelength components so that different members of theset or different portions of the continuous distribution impinge uponcorresponding ones of the light sensing elements. The one or moreoptical devices may include devices such as diffraction gratings,prisms, optical filters, mirrors, etc. In one embodiment, the one ormore optical devices include a diffraction grating.

In one embodiment, the light sensing device 640 may include three lightsensing elements for three-channel color measurement (e.g., RGBmeasurement), with each light sensing element capturing a correspondingone of the three colors. In another embodiment, the light sensing device640 may include two or more light sensing elements corresponding todifferent subbands in the infrared band.

In one embodiment, the light sensing device 640 may be (or include) aspectrometer.

In some embodiments, the light sensing device 640 may be (or include) amotion sensor 640M, e.g., as illustrated in FIG. 8. The motion sensormay be (or include) any of a wide variety of known devices for motionsensing or motion detection. The motion sensor is configured to generatethe data stream D₂ in response to the light stream S₂. For example, themotion sensor may generate an analog electrical signal in response tothe light stream S₂ and digitize the analog electrical signal in orderto obtain a sequence of samples of the analog electrical signal. When anobject enters into the field of view, the analog electrical signal andthe data stream D₂ (the sequence of samples) exhibit a disturbance. Theabove-described light variation event may be interpreted as being thisdisturbance. The control unit may detect the disturbance by detectingthe occurrence of a pulse of sufficient amplitude in the analogelectrical signal or in the data stream D₂. In one embodiment, thecontrol unit 650 may detect the disturbance by comparing the absolutevalue of the sampled signal (the sequence of samples) or its timederivative to a threshold. The light variation event occurs when thethreshold is exceeded. (Computational algorithms for detectingmotion-induced disturbances in the signal(s) generated by motion sensorsare well known in the prior art, and thus, need not be elaborated here.)

In some embodiments, the motion sensor 640M may include built-incircuitry for detecting the disturbance. In these embodiments, thebuilt-in circuitry may assert a signal in response to the detection ofthe disturbance and inject the signal into the data stream D₂. Thecontrol unit 650 recognizes that the disturbance has been detected whenit receives the signal from the data stream D₂.

Objects of interest may be known to emit light (electromagnetic energy)in a particular wavelength band. Thus, in some embodiments, the motionsensor 640M may be sensitive to light in that wavelength band. Forexample, in one embodiment, the motion sensor may include a passiveinfrared sensor. Various objects of interest (human beings, automobiles,aircraft, buildings, etc.) are known to emit infrared radiation. Thus,the entrance of such an object into the field of view will beaccompanied by a disturbance (e.g., a step) in the infrared band.

In some embodiments, system 600 may include a motion sensor 645 inaddition to the light sensing device 640, e.g., as shown in FIG. 9. Inthese embodiments, the optical subsystem 610 may be configured toseparate the incident light stream L into three output streams includingthe light stream S₁, the light stream S₂ and the light stream S₃. Asdescribed above, the light stream S₁ is provided to the light modulationunit 620, and the light stream S₂ is provided to the light sensingdevice 640. The light stream S₃ is provided to the motion sensor 645.The motion sensor 645 is configured to receive the light stream S₃ andgenerate a sequence of samples D₃ in response to the light stream S₃(e.g., similar to what is described above in connection with motionsensor 640M). The control unit 650 may be configured to monitor thesequence of samples D₃, e.g., to detect a motion-induced disturbance inthe sequence of samples. The above-described trigger condition (used toturn on power to the light modulation unit 620) may include thecondition of detecting this motion-induced disturbance and the conditionof detecting the light variation event. By combining the two detectionconditions, the system 600 may be more immune to false alarms. (A falsealarm is the situation where the system decides to turn on power to thelight modulation unit when there is actually no object of interest inthe field of view.)

The optical subsystem 610 may separate the incident light stream L intothree output streams (S₁, S₂ and S₃) in any of various ways, using anycombination of optical devices. The streams S₁, S₂ and S₃ may bespectrally similar or dissimilar to the incident light stream L.Furthermore, the streams S₁, S₂ and S₃ may be spectrally similar ordissimilar to each other. For example, in one embodiment, the streams S₂and S₂ are restricted to the infrared band, while stream S₁ includeslight in the visible spectrum.

In some embodiments, the light sensing device 630 may include a singlelight sensing element and an analog-to-digital converter (ADC), e.g., asdescribed above in connection with light sensing device 130 and ADC 140.The light sensing element may be configured to generate an analogelectrical signal representing intensity of the modulated light streamMLS as a function of time. For example, the light sensing element may bea photodiode. The ADC is configured to capture a sequence of samples ofthe analog electrical signal. (Each sample represents the intensity ofthe modulated light stream at a corresponding time.) The data stream D₁may include this sequence of samples.

In some embodiments, the light sensing device 630 may include aplurality (e.g., an array) of light sensing elements, each configured toreceive a corresponding spatial portion of the modulated light streamMLS. For each of the light sensing elements, the light sensing device630 is configured to capture a corresponding sequence of intensitysamples representing intensity over time of the corresponding spatialportion. The data stream D₁ may include these sequences of intensitysamples captured from the light sensing elements. When imagereconstruction is performed on the data stream D₁, the sample sequencecaptured from each light sensing element may be used to reconstruct acorresponding subimage that is representative of a correspondingsubregion of the scene under observation. A complete image, representingthe whole of the scene under observation, may be generated by joiningtogether (concatenating) the subimages. For more information on how toreconstruct an image (or sequence of images) from a data stream obtainedfrom a set of parallel light sensing elements, please see U.S. patentapplication Ser. No. 13/197,304, filed on Aug. 3, 2011, entitled“Decreasing Image Acquisition Time for Compressive Imaging Devices”.

In some embodiments, the light sensing device 630 may be configured asvariously described in U.S. patent application Ser. No. 13/197,304. Forexample, light sensing device 630 may include the “light sensing device630” and the “sampling subsystem 640” described in that patentapplication.

In some embodiments, the light sensing device 630 may include aplurality of light sensing elements, each configured to receive acorresponding spectral portion of the modulated light stream MLS. Foreach of the light sensing elements, the light sensing device 630 isconfigured to capture a corresponding sequence of intensity samplesrepresenting intensity over time of the corresponding spectral portionof the modulated light stream. The data stream D₁ may include thesesequences of intensity samples captured from the light sensing elements.Thus, the data stream D₁ may be interpreted as a sequence of spectralintensity vectors. The elements of each vector may represent theintensity of corresponding wavelength components or wavelength bands inthe modulated light stream MLS.

One or more optical devices may intervene along the optical path betweenthe light modulation unit 620 and the light sensing device 630 in orderto spatially separate the modulated light stream MLS into a set orcontinuous distribution of wavelength components so that differentmembers of the set or different portions of the continuous distributionimpinge upon corresponding ones of the light sensing elements. Forexample, the one or more optical devices may include devices such asdiffraction gratings, prisms, optical filters, mirrors, etc.

In some embodiments, the light sensing device 630 may be (or include) aspectrometer.

In some embodiments, the control unit 650 may be configured to monitorthe data stream D₁ in order to detect a spectral signature of interestin the modulated light stream MLS. For example, in a system targeted forthe observation of vehicles, the control unit may monitor the datastream D₁ for spectral signatures characteristic of the exhaust gasesgenerated by one or more kinds of vehicles. In a system targeted for thesurveillance of human beings, the control unit may monitor the datastream for the presence of infrared emissions or certain characteristicpatterns of IR emission. In a system targeted for chemical plumedetection, the control unit may search the data stream for the presenceof any of a predetermined set of spectral patterns corresponding to oneor more chemical plumes of interest. In a system targeted forastronomical observation, the control unit may monitor the data streamfor the presence of a spectral pattern of interest to the user, e.g.,the emission spectra of certain elements or combinations of elements.

While monitoring the data stream D₁ for a spectral signature ofinterest, the control unit 650 may direct the light modulation unit 620to apply one or more spatial patterns having all one values, i.e.,spatial patterns where all the light reflecting elements are set to theorientation state that reflects light to the light sensing device 630.(In some embodiments, the spatial patterns may be patterns that take theone value within a subregion of the field of view and the zero valueoutside that subregion.) Alternatively, the control unit may direct thelight modulation unit to apply spatial patterns from a measurementpattern set, i.e., a pattern set that is incoherent relative to thesparsity pattern set in which the signal is compressible (or sparse),thus enabling the reconstruction of a separate image for each spectralportion of the modulated light stream from the corresponding sequence ofintensity samples.

In some embodiments, the spectral signature detection may be performedusing a prior art detection algorithm.

In some embodiments, the control unit may be configured to execute areconstruction algorithm in response to detecting the spectral signatureof interest, e.g., as variously described above. The reconstructionalgorithm may be configured to compute an image or image sequence (thatrepresents the external scene) based on at least a subset of the datastream D₁. The “subset” of the data stream D₁ may be a subset startingat the time the spectral signature of interest is detected, or, apredetermined amount of time before the spectral signal detection. (Thelatter option may be implemented by buffering the data stream D₁ inmemory, thus allowing access to past values of the data stream D₁.)

The control unit 650 may inject measurement patterns into the sequenceof spatial patterns to be used by the light modulation unit 620 inresponse to detecting the spectral signature of interest. If the controlunit has already been injecting measurement patterns while monitoringfor the spectral signature of interest, it may continue to do so afterdetecting that spectral signature. The “subset” of the data stream D₁used to perform the reconstruction is preferably a subset thatcorresponds to the measurement patterns (although not necessarily acontiguous subset in time since other kinds of patterns may beinterspersed with the measurement patterns in the sequence of spatialpatterns).

An image reconstructed by the control unit 650 may include a pluralityof component images corresponding respectively to a plurality wavelengthbands. Each component image is reconstructed from a subset of theintensity samples captured from a corresponding one of the light sensingelements, and represents a spectrally-limited view of the externalscene, i.e., a view that is limited to the portion of theelectromagnetic spectrum captured by the corresponding light sensingelement. (Recall that each light sensing element receives acorresponding spectral portion of the modulated light stream.) Forexample, in the case where the modulated light stream is separated intored, green and blue components, and the light sensing device has threelight sensing elements for respectively capturing those threecomponents, the control unit may reconstruct a multi-spectral imagehaving three component images, i.e., a red image based on samples fromthe red light sensing element, a green image based on samples from thegreen light sensing element, and a blue image based on samples from theblue light sensing element. This example naturally extends to any numberof components and any set of wavelength bands. The multispectral imagemay have any number of components images, depending on the number oflight sensing elements.

In some embodiments, the system 600 may also include a transmitter,e.g., as variously described above. The control unit 650 may beconfigured to direct the transmitter to transmit at least a subset ofthe data stream D₁ onto a communication channel. The subset may be asvariously discussed above.

In some embodiments, the control unit 650 may be configured to searchfor a region within the field of view that contains a bright object (oran object that is bright than the general background) such as the sun ora reflection of the sun from a shiny surface in the field of view. Thesearch may be performed after turning on power to the light modulationunit 620. The search may include injecting a sequence of search patternsinto the sequence of spatial patterns, and selecting a next searchpattern to be injected into the sequence of spatial patterns based on aportion of the data stream D₁ that corresponds to one or more previousones of the search patterns that have already been injected into thesequence of spatial patterns. (For more information on how to conductsuch a search, please refer to U.S. patent application Ser. No.13/207,276, filed on Aug. 10, 2011, entitled “Dynamic Range Optimizationin a Compressive Imaging System”, which is hereby incorporated byreference in its entirety.) Once the region corresponding to the brightobject has been determined, the control unit 650 may mask out (i.e.,remove) the bright object by masking the spatial patterns supplied tothe light modulation unit 620. In particular, each of the spatialpatterns may be masked so that it is set to zero (or perhaps,attenuated) within the bright object region but unmodified outside thebright object region. Thus, portions of the modulated light stream MLScorresponding to the bright object region do not reach (or reach withattenuated intensity) the light sensing device 630 while portions of themodulated light stream corresponding to the exterior of the brightobject region are modulated as they would have been without the masking.This masking process may be applied to a sequence of measurementspatterns. The subset of the data stream D₁ corresponding to the maskedmeasurement patterns may be used to reconstruct an image (or imagesequence) corresponding to the exterior of the bright object region.

In one alternative embodiment, the control unit may mask out (i.e.,remove) the set complement of the bright object region, again by maskingthe spatial patterns. In particular, each of the spatial patterns is setto zero (or perhaps, attenuated) outside the bright object region andunmodified inside the bright object region. Thus, portions of themodulated light stream outside the bright object region do not reach thelight sensing device 630, while portions of the modulated light streaminside the bright object region are modulated as they would have beenwithout the masking. This masking process may be applied to a sequenceof measurements patterns. The subset of the data stream D₁ correspondingto the masked measurement patterns may be used to reconstruct an image(or image sequence) corresponding to the interior of the bright objectregion.

In some embodiments, system 600 may also include an audio sensor such asa microphone or an array of microphones. The control unit 650 may beconfigured to monitor the signal(s) generated by the audio sensor todetect an audio event of interest, e.g., an occurrence of an audiofeature or signal of interest. In one embodiment, the control unit maydetect an increase in audio power. In another embodiment, the controlunit may detect an occurrence of an audio spectrum (or spectrogram)belonging to a signal class of interest, e.g., an audio spectrum (orspectrogram) characteristic of the human voice. The detection of theaudio event may be used to further qualify the trigger condition.(Recall that the trigger condition determines whether to turn on powerto the light modulation unit 620). For example, the control unit mayrequire both the light variation event and the audio event occur inorder to assert the trigger condition.

In some embodiments, the system 600 may include one or more sensors suchas chemical sensors, pressure sensors, proximity sensors, radiationsensors, particle detectors (e.g., Geiger counters), smoke detectors andvibration sensors. The sensing signals generated by any combination ofsuch sensors may be used to qualify the trigger condition.

In some embodiments, the system 600 may include a light source forilluminating the external scene. For example, the light source may bepowered on in response to detection of the light variation event.

In some embodiments, the optical subsystem 610 may be (or include) a TIRprism pair 610T, e.g., as shown in FIG. 10. (TIR is an acronym for“total internal reflection”.) The TIR prism pair splits the incidentlight stream L into the light streams S₁ and S₂ at one or more of itsinternal faces, i.e., the faces where the two prisms come into closeproximity. The light stream S₁ proceeds to the light modulation unit 620where it experiences modulation with the time sequence of spatialpatterns. The modulated light stream MLS returns to the TIR prism pairwhere it experiences total internal reflection at one of the internalfaces. After total internal reflection, the modulated light stream exitsthe TIR prism pair onto an optical path leading to the light sensingdevice 630.

In other embodiments, the optical subsystem 610 may be (or include) adual TIR prism.

For more information on how to use TIR prisms and dual TIR prisms incompressive imaging systems, please refer to U.S. patent applicationSer. No. 13/207,900, filed on Aug. 11, 2011, entitled “TIR Prism toSeparate Incident Light and Modulated Light in Compressive ImagingDevice”, which is hereby incorporated by reference in its entirety.

In some embodiments, the system embodiment of FIG. 9 may be realized asshown in FIG. 11, where the optical subsystem 610 includes a beamsplitter 610S and the TIR prism pair 610T. The beam splitter 610S splitsin the incident light stream L into the light stream S₂ and anintermediate stream K. The TIR prism pair 610T splits the intermediatestream K into the light stream S₁ and the light stream S₃. The lightstream S₁ proceeds to the light modulation unit 620 while the lightstream S₃ proceeds to the motion sensor 645.

In some embodiments, the system 600 may be configured as shown in FIG.12. In these embodiments, the modulated light stream MLS may beseparated by an optical subsystem 625 into light stream T_(A) and T_(B).The light stream T_(A) is supplied to a light sensing device 630A; andthe light stream T_(B) is supplied to a light sensing device 630B. Lightsensing device 630A generates a data stream D_(A) in response to thelight stream T_(A), e.g., as variously described above in connectionwith light sensing device 630. Similarly, light sensing device 630Bgenerates a data stream D_(B) in response to the light stream T_(B),e.g., as variously described above in connection with light sensingdevice 630. The control unit 650 may monitor the data stream D₂ anddetermine when to turn on power to the light modulation unit 620. Thecontrol unit 650 may also monitor the data stream D_(A) and/or the datastream D_(B) in order to decide when to invoke reconstruction on thedata stream D_(A) and/or the data stream D_(B), or when to invoketransmission of the data stream D_(A) and/or the data stream D_(B).

In some embodiments, light sensing device 630A may include one or morelight sensing elements, each configured to convert a correspondingspatial portion of the light stream T_(A) into a corresponding sequenceof intensity samples; and light sensing device 630B may include one ormore light sensing elements, each configured to convert a correspondingspectral portion of the light stream T_(B) into a corresponding sequenceof intensity samples.

Event Detection Using a Compressive-Sensing Hyperspectral-ImagingArchitecture

In one set of embodiments, a compressive imaging system 1300 may beconfigured as shown in FIG. 13. The system may be configured to detectevents occurring within a field of view, e.g., momentary events such asexplosions, gun discharges, chemical reactions, launches of rockets,missiles, rocket-propelled grenades, etc. In addition, the system maydetermine the spatial locations of the events within the field of view,and highlight those locations in an image sequence (e.g., a videosequence) representing the field of view so a user can perceive theevent in its visual context.

The system 1300 may include a spectral separation subsystem 1310, anarray 1315 of light sensing elements, a sampling subsystem 1320 and adetection unit 1325. (Furthermore, the system 1300 may include anysubset of the features, embodiments and elements discussed above withrespect to system 100, system realization 200 and FIGS. 6-12, anddiscussed below with respect to method 1700, system 1800, system 2400,system 2800 and system 2900.)

The spectral separation subsystem 1310 may be configured to receive amodulated light stream (MLS), where the modulated light stream isgenerated by modulating an incident light stream with a temporalsequence of spatial patterns, e.g., measurement patterns as variouslydescribed above. The modulation may be performed using a lightmodulation unit as variously described above. The spectral separationsubsystem may be further configured to separate the modulated lightstream into a plurality of wavelength components, e.g., so thatdifferent wavelengths components have different spatial positions. (FIG.13 highlights two of the wavelength components, i.e., wavelengthcomponents λ_(a) and λ_(b), not to suggest a limitation on the number ofwavelength components, but to illustrate the principle that differentwavelength components have different spatial positions within thebeam(s) outputted from the spectral separation subsystem.) In someembodiments, the spectral separation subsystem may include one or morediffraction gratings, or one or more prisms, or one or more spectralfilters, or any combination of the foregoing types of element.

In some embodiments, the “plurality of wavelengths components” producedby the spectral separation subsystem comprise a continuous distributionof wavelength components. In other words, the wavelength components arespread out spatially in a continuous fashion. In other embodiments, theplurality of wavelength components may comprise a set of discretewavelength bands, e.g., such as might result from filtering themodulated light stream with a series of spectral filters.

The light sensing elements of the array 1315 may be configured toreceive respective subsets of the wavelength components and to generaterespective signals (e.g., analog signals). Each of the signalsrepresents intensity of the respective subset of the wavelengthcomponents as a function of time. In other words, each light sensingelement produces a signal representing the instantaneous intensity ofthe wavelength components impinging upon its sensing surface. The lightsensing elements may be realized as variously described above, e.g., inconnection with system 100 and system realization 200. For example, thelight sensing elements may be photodiodes.

In some embodiments, the light sensing array 1315 may be aone-dimensional array, e.g., a linear array, or alternatively, anon-uniformly spaced array. In other embodiments, the array may be atwo-dimensional array. In some embodiments, the light sensing elementsof the array 1315 may be implemented as part of a single integratedcircuit. Alternatively, the array 1315 may be formed from a plurality ofseparately packaged devices.

The sampling subsystem 1320 may be configured to sample the signals inorder to obtain respective sample sequences. (The sample sequences maybe stored into a memory of system 1300.) The sampling subsystem mayinclude a plurality of analog-to-digital converters (ADCs), e.g., thesame number of ADCs as light sensing elements in the array 1315.However, in some embodiments, the sampling subsystem may include fewerADCs by employing time multiplexing. For example, two or more of thesignals may be sampled by a single ADC by employing a multiplexer torapidly alternate (or cycle among) the two or more signals.

The detection unit 1325 may be configured to monitor a selected subsetof the signals to detect an event occurring within a field of viewcorresponding to the incident light stream. (The selected subset may beprogrammable, e.g., by a host computer external to the system 1300,and/or, via a user interface provided as part of the system 1300.) Bysaying that the detection unit monitors a selected subset of “thesignals”, we do not mean to suggest that the detection unit must in allcases operate directly on analog outputs from the array 1315 as shown inFIG. 13. It is also possible that the detection unit may operate on aselected subset of the sample sequences, as shown in FIG. 14. Bothrealizations are to be interpreted as falling within the scope ofmeaning of the phrase “monitoring a selected subset of the signals”produced by the light sensing array 1315.

The action of detecting the event may include determining when theselected subset of signals satisfy a pre-determined signal condition.The nature of the pre-determined signal condition may be different indifferent embodiments. In some embodiments, the pre-determined signalcondition is the condition that the signals of the selected subsetsimultaneously exceed respective programmable thresholds:y _(f(i))(t)>T _(i) ,i=0,1,2, . . . ,N _(SS)−1,where {y(t): j=1, 2, . . . , N_(LSE)} denotes the signals generatedrespectively by the light sensing elements of the array 1315, whereN_(LSE) denotes the number of light sensing elements, where f denotes aselection function that identifies the signals belonging to the selectedsubset, where f(i) represents the index of the i^(th) signal of theselected subset, where N_(SS) is the number of signals in the selectedsubset, where T_(i) is the i^(th) threshold. The detection unit 1325 mayinclude a plurality of analog comparator circuits configured to comparerespective signals of the selected subset to respective programmablethresholds. Alternatively, the detection unit may include a plurality ofdigital comparator circuits. Each of the digital comparator circuits maybe configured to compare a respective one of the sample sequences of theselected subset to a respective programmable threshold.

In some embodiments, the detection unit may include a plurality ofdigital circuits, where each of the digital circuits is configured tocompare a rate of change of a respective one of the sample sequences ofthe selected subset to a respective programmable threshold:d{y _(f(i))(t)}/dt>R _(i) ,i=0,1,2, . . . ,N _(SS)−1.Alternatively, the detection unit may include a plurality of analogcircuits, where each of the analog circuits is configured to compare arate of change of a respective one of the signals of the selected subsetto a respective programmable rate threshold.

In some embodiments, the pre-determined signal condition is the logicalAND of a first condition and a second condition, where the firstcondition is that the signals of the selected subset simultaneouslyexceed respective programmable value thresholds, and the secondcondition is that rates of change of the respective signals of theselected subset simultaneously exceed respective programmable ratethresholds.

In some embodiments, the pre-determined signal condition is thecondition that the signals of the selected subset have respective valuesthat conform to a pre-determined spectral signature. For example, thedetection unit may monitor the vector signal <y_(f(i)))(t): i=0,1, . . .,N_(SS)> to determine when the vector signal conforms to (i.e., matches)a predetermined spectral signature vector S=<s_(i):i=0, 1, . . .,N_(SS)>. The spectral signature vector may be programmable. In someembodiments, the detection unit may monitor the vector signal todetermine when vector signal matches any spectral signature in apredetermined set of spectral signals, e.g., spectral signatures ofdifferent types of explosions, chemical reactions, firearm discharges,etc.

In some embodiments, the system 1300 may also include a processing unit1330, e.g., as shown in FIG. 15A or FIG. 15B. The processing unit may berealized by one or more processors executing program instructions, bycustom-designed digital circuitry such as one or more ASICs, by one ormore programmable hardware elements such as FPGAs, or by any combinationof foregoing types of processing element.

The processing unit 1330 may be configured to receive the samplessequences produced by the sampling subsystem 1320 and to reconstruct atemporal sequence (e.g., a video sequence) of images based on the samplesequences. Each image of the temporal sequence of images may bereconstructed based on a corresponding window of samples. (The samplewindows may be overlapped in time, e.g., as variously described in U.S.patent application Ser. No. 13/534,414. In other words, each samplewindow may overlap the previous sample window by a certain amount.) Theprocessing unit may direct the temporal sequence of images to bedisplayed, e.g., by invoking a display process that transfers thetemporal sequences of images to a display device.

The temporal sequence of images may be generated in different ways indifferent embodiments. In some embodiments, the sample sequences may beadded together to obtain a composite sequence, and the images of thetemporal sequence may be reconstructed based on correspondingoverlapping windows of samples from the composite sequence. The processof adding the sample sequences may be modeled by the expression:

${{s(k)} = {\sum\limits_{j = 0}^{N_{LSE}}{y_{j}(k)}}},$where k is a discrete time index. In one embodiment, the samplesequences may be added with weighting. In other words, the compositesequence may be a linear combination of the sample sequences.

In one alternative embodiment, the sample sequences may be partitionedinto three groups, with each group corresponding to a distinct band ofwavelengths. For example, the three groups may correspond respectivelyto a short wavelength band, a medium wavelength band and a longwavelength band. The sample sequences in each group may be summed toobtain a corresponding sum sequence. Thus, three sum sequences s_(S)(k),s_(M)(k) and s_(L)(k) are generated. In each frame time, a color imagemay be generated by reconstructing a blue sub-image from a window ofsamples of the sum sequence s_(S)(k), reconstructing a green sub-imagefrom a window of samples of the sum sequence s_(M)(k), andreconstructing a red sub-image from a window of samples of the sumsequence s_(L)(k). The resulting sequence of color images may bedisplayed using a color display.

In some embodiments, the processing unit 1330 may be configured toreconstruct a first image and a second image in response to thedetection of the event. (The processing unit 1330 may receive an eventdetection signal from the detection unit when the event has beendetected.) The first image is reconstructed based on a first window ofsamples, and the second image is reconstructed based on a second windowof samples. Each window of samples is taken from (or derived from) thesample sequences corresponding to the selected subset. However, thesecond window is advanced in time relative to the first window. Thefirst window may correspond to a first time interval prior to the event,e.g., a time interval ending just prior to the event. The second windowmay correspond to a second time interval that at least partiallyincludes the event. The first window and the second window may overlapin time (e.g., as variously described in U.S. patent application Ser.No. 13/534,414). In some embodiments, the first window and the secondwindow overlap in time by a high percentage. For example, the secondwindow may have the same number of samples as the first window but beshifted forward in time by an amount Δt which ensures that the secondwindow includes a substantial portion of the event duration. (Theprocessing unit may derive a measure of the event duration by measuringthe duration of assertion of the event detection signal.) In someembodiments, the reconstructed first image may be used as a warm startfor the reconstruction of the second image, e.g., as variously describedin U.S. patent application Ser. No. 13/534,414.

In some embodiments, the processing unit 1330 is further configured todetermine spatial localization information based on a difference betweenthe first image and the second image: ΔI=I₂−I₁. The spatial localizationinformation indicates where the event has occurred in the field of view.The computation of the difference image may serve to remove the commonbackground that is shared by the first and second images, leaving energycorresponding to the event. Thus, most of the pixels in the differenceimage may be close to zero except for the pixels corresponding to theevent. In one embodiment, the spatial localization information may bedetermined by computing a centroid and spatial radius of the pixels inthe difference image. The spatial radius may be computed by averagingthe distance (or the squared distance) between each pixel position(j_(x), j_(y)) and the centroid. The distance value for each pixel maybe weighted by the value of the pixel. In another embodiment, theprocessing unit may apply a threshold to the pixel amplitudes of thedifference image, and then compute a minimal bounding box (or circle orother geometric shape) that contains the pixels which exceeded thethreshold. After determining the spatial localization information, theprocessing unit may inject (e.g., overlay, blend or superimpose) avisual representation of the spatial localization information into atleast a subset of the images of said temporal sequence of images, e.g.,as shown in image 1610 of FIG. 16. A circle 1615 with cross hairs (orsome other type of visual indication) may be injected into the image toindicate the location and approximate extent of the event 1620 (e.g., anexplosion, discharge, gun blast, rocket launch or missile launch). Inone embodiment, the visual representation may be injected into images ofthe temporal sequence starting at the time of the event detection, orperhaps slightly before the event detection time.

The spatial localization information may include a location ofoccurrence of the event within the field of view. Furthermore, thespatial localization information may include a subregion (e.g., a diskor ellipse or rectangle or convex polygon) that spatially bounds orcontains the event within the field of view.

In some embodiments, the system 1300 may be realized as a camera. Thecamera may be mounted on a platform that allows angular (e.g., azimuthand elevation) adjustments. Furthermore, the optical input path (wherebythe incident light stream enters the camera and is supplied to the lightmodulation unit) may include a camera lens subsystem that has a range ofoptical zoom. The processing unit 1330 may be configured to direct oneor more actuators to adjust a pointing direction and/or extent ofoptical zoom of the camera based on the spatial localizationinformation. For example, it may be desirable to reorient the camera sothat the event location is centered in the field of view, and/or, sothat the spatial region associated with the event occupies a higherpercentage of the field of view.

In some embodiments, the processing unit 1330 may be configured toreconstruct an image I_(E) in response to the detection of the event.The image I_(E) may be reconstructed based on a window W_(E) of samplestaken (or derived) from the sample sequences corresponding to theselected subset. The window W_(E) corresponds to a time interval that atleast partially includes the event, e.g., includes the entirety of theevent or an initial portion of the event. The processing unit 1330 maydetermine spatial localization information based on the image I_(E).(The spatial localization information indicates where the event hasoccurred in the field of view.) For example, the processing unit 1330may apply a threshold to the pixels of the image I_(E), and then computea centroid and radius of the surviving pixels (i.e., the pixels thatexceeded the threshold). Instead of a centroid and radius, theprocessing unit may alternatively compute a minimal bounding boxcontaining the event. Under the assumption that the pixels correspondingto the event are brighter than the pixels of the background, theapplication of the threshold may eliminate most of the background,leaving pixels corresponding to the event. After having determined thespatial localization information, the processing unit may inject avisual representation of the spatial localization information into atleast a subset of the images of said temporal sequence of images, e.g.,as described above.

In some embodiments, the processing unit 1330 may be configured toperform a search process in response to the detection of the event. Thesearch process may operate on one or more of the sample sequencesbelonging to the selected subset, and during the occurrence of theevent, in order to identify a spatial subregion within the field of viewthat contains the event. The search process may include: injectingsearch patterns into the temporal sequence of spatial patterns; andanalyzing the samples of the one or more sample sequences in response tothe injection of the search patterns. The search process is adaptive inthe sense that the analysis of previous samples guides the selection ofnew search patterns to inject into the temporal sequence of spatialpatterns. (For example, if a first search pattern corresponding to afirst spatial region in the field of view produces a sample with highintensity value, additional search patterns corresponding tonon-overlapping subregions of the first region may be injected into thesequence of spatial patterns, to explore within the first region.) Thesearch process may be conducted according to any of the various methodsdescribed in U.S. patent application Ser. Nos. 13/631,626 and13/207,276. After having identified the spatial subregion containing theevent, the processing unit 1330 may inject a visual representation ofthe spatial subregion into at least a subset of the images of saidtemporal sequence of images, e.g., as described above.

In some embodiments, the search process includes a hierarchical searchbased on a quadtree, e.g., as described in U.S. patent application Ser.No. 13/631,626. The quadtree corresponds to a recursive partitioning ofthe field of view (i.e., the array of light modulating elements) intorectangular subsets.

In one set of embodiments, a method 1700 may include the operationsshown in FIG. 17. The method 1700 may be used to detect an event (suchas an explosion, a gun discharge or chemical reaction) occurring withina field of view based on signal measurements made over a plurality ofspectral channels. (The method 1700 may also include any subset of thefeatures, elements and embodiments described above in connection withsystem 100, system realization 200, system 1300 and FIGS. 6-16, anddescribed below in connection with system 1800, system 2400, system 2800and system 2900.)

At 1710, a modulated light stream may be received. The modulated lightstream may be generated by modulating an incident light stream with atemporal sequence of spatial patterns, e.g., as variously describedabove in connection with system 100, system realization 200 and system1300.

At 1715, the modulated light stream may be separated into a plurality ofwavelength components, e.g., as variously described above in connectionwith spectral separation subsystem 1310.

At 1720, subsets of the wavelength components may be converted intorespective signals, e.g., as variously described above. Each of thesignals represents intensity of the respective subset of the wavelengthcomponents as a function of time.

At 1725, the signals are sampled in order to obtain respective samplesequences, e.g., using the sampling subsystem 1320 described above. Thesample sequences may be stored into a memory, e.g., to allow subsequentreconstruction of images.

At 1730, a selected subset of the signals are monitored to detect anevent occurring within a field of view corresponding to the incidentlight stream. The action of detecting the event includes determiningwhen the selected subset of signals satisfy a pre-determined signalcondition. The selected subset may be programmable. Thus, in differentcontexts, different subsets of the signals may be used.

In some embodiments, the pre-determined signal condition is thecondition that the signals of the selected subset simultaneously exceedrespective programmable thresholds. In other embodiments, thepre-determined signal condition is the condition that rates of change ofthe respective signals of the selected subset exceed

respective programmable thresholds. In yet other embodiments, thepre-determined signal condition is the condition that the signals of theselected subset have respective values that conform to a pre-determinedspectral signature.

In some embodiments, the method 1700 may also include reconstructing atemporal sequence of images based on the sample sequences. In oneembodiment, the temporal sequence of images may be reconstructedcontinuously, e.g., as along as the sample sequences are beinggenerated. In another embodiment, the reconstruction of the temporalsequence of images is initiated in response to the detection of theevent. The above-described storage of the sample sequences into thememory allows images corresponding to past history to be reconstructedif desired.

In some embodiments, the method 1700 also includes reconstructing afirst image and a second image in response to the detection of theevent. The first image is reconstructed based on a first window ofsamples, and the second image is reconstructed from a second window ofsamples. Each window of samples is taken (or derived) from the samplesequences corresponding to the selected subset. The first windowcorresponds to a first time interval prior to the event, and the secondwindow corresponds to a second time interval that at least partiallyincludes the event.

After having reconstructed the first image and the second image, themethod 1700 may determine spatial localization information based on adifference between the first image and the second image (where thespatial localization information indicates where the event has occurredin the field of view), and inject a visual representation of the spatiallocalization information into at least a subset of the images of thetemporal sequence of images.

In some embodiments, the method 1700 may also include reconstructing animage I_(E) in response to the detection of the event. Thereconstruction of the image I_(E) is based on a window W_(E) of samplestaken (or derived) from the sample sequences corresponding to theselected subset. The window W_(E) of samples corresponds to a timeinterval that at least partially includes the event. After havingreconstructed the image I_(E), spatial localization information may bedetermined based on the image I_(E), e.g., as described above inconnection with processing unit 1330. The spatial localizationinformation indicates where the event has occurred in the field of view.Furthermore, a visual representation of the spatial localizationinformation may be injected into at least a subset of the images of thetemporal sequence of images.

In some embodiments, the method 1700 may include performing a searchprocess in response to the detection of the event. The search processmay operate on one or more of the sample sequences belonging to theselected subset and during the occurrence of the event, in order toidentify a spatial subregion of the field of view that contains theevent. The search process may include: injecting search patterns intothe temporal sequence of spatial patterns; and analyzing the samples ofthe one or more sample sequences in response to the injection of thesearch patterns. In one embodiment, a visual representation of thespatial subregion may be injected into at least a subset of the imagesof said temporal sequence of images.

In some embodiments, the search process includes a hierarchical searchbased on a quadtree, where the quadtree corresponds to a recursivepartitioning of the field of view into rectangular subsets.

In one embodiment, the spectral separation subsystem and the array oflight sensing elements may be incorporated as part of a commercialoff-the-shelf spectrometer.

Compressive-Sensing Hyperspectral Imaging System

In one set of embodiments, a compressive sensing (CS) hyperspectralimaging system 1800 may be configured as shown in FIG. 18. The system1800 may capture a stream of compressive measurements for each of aseries of wavelength bands (spectral windows) covering a wavelengthspectrum, e.g., the infrared spectrum, or the short-wave infraredspectrum, or the UV spectrum, or the visible spectrum, or a broadspectrum including both the SWIR spectrum and the visible spectrum, etc.For each wavelength band, the system may reconstruct a correspondingcomponent image based on the corresponding stream of compressivemeasurements. The component image represents the field of view (theexternal scene) restricted to the corresponding wavelength band. Thecomponents images together form a hyperspectral data cube.

The system 1800 may include a digital micromirror device (DMD) thatmodulates an incident light stream with a temporal sequence of spatialpatterns to obtain a modulated light stream MLS. The spatial patternsinclude measurements patterns, i.e., patterns that are incoherentrelative to the sparsity pattern set being assumed for the incidentlight stream, as variously described above.

A lens 1810 may be used to image, focus or direct the modulated lightstream onto a diffraction grating 1815. (The diffraction grating 1815may be reflection grating or a transmission grating.) The diffractiongrating spatially separates the wavelength components contained in themodulated light stream. While FIG. 18 highlights four of the wavelengthcomponents (using different types of line texture), the light streamoutputted from the diffraction grating may include any number ofwavelength components, e.g., an infinite continuum of wavelengthcomponents. The outputted light stream may be focused or imaged ontosensing and detection subsystem 1825, e.g., using a curved mirror 1820,or alternatively, using a second lens.

The subsystem 1825 includes an array of light sensing elements, each ofwhich receives a corresponding sub-band of the wavelength components inthe outputted light stream and generates a corresponding intensitysignal that represents intensity as a function of time of thecorresponding sub-band of wavelength components. The subsystem 1825 mayalso include an array of analog-to-digital converters (ADCs) configuredto convert the sub-band intensity signals into respective sub-bandsample sequences 1830 in parallel. Each of the sub-band sample sequencesmay be used to reconstruct a corresponding sub-band image (or sub-bandimage sequence). The set 1835 of sub-band images form a 3Drepresentation of the external scene extending in two spatial dimensionsand in the wavelength dimension. (The set of sub-band image sequencesform a 4D representation of the external scene extending in two spatialdimensions and in the wavelength and time dimensions.)

In some embodiments, the system 1800 is configured to very quicklyrecognize a short-duration spectral signature, and then identify thespatial location of the source of the spectral signature within thefield of view. To facilitate such recognition, the subsystem 1825 mayinclude a detection unit, e.g., as variously described above inconnection with system 1300.

Continuous High-Speed Spectral Signature (SS) Detection

In some embodiments, the sensing and detection subsystem 1825 mayinclude an array 1910 of photodiodes and a plurality of photodiodemonitoring blocks (PMBs). Each PMB may couple to a respective one of thephotodiodes of the photodiode array. In some embodiments, the sensingand detection subsystem 1825 may be realized by a custom-designed ASIC.

Each PMB may include high-speed dedicated circuitry configured tocontinuously monitor the output of the respective photodiode in order todetect abrupt changes in light energy in the corresponding spectralwindow, i.e., the spectral window of wavelength components received bythe respective photodiode. (The threshold levels used by the PMBs arealso programmable.) Each PMB may monitor the output signal generated bythe respective photodiode and compare the instantaneous amplitude(and/or the instantaneous rate of change of the amplitude) of the outputsignal to a respective threshold. When the threshold is exceeded, thePMB may send a detection signal to the system controller block (SCB).(The SCB may be implemented, e.g., by the above-described processingunit 1330.) The SCB may monitor the detection signals from a selectedsubset of the PMBs, e.g., PMBs corresponding to spectral windows ofinterest. When the detection signals indicate that all the spectralwindows of interest have exceeded their respective thresholds, aspectral signature (SS) event is declared.

FIG. 20 shows one embodiment of the photodiode monitoring block (PMB). Atransimpedance amplifier (TIA) 2010 converts the current output of therespective photodiode into a voltage signal. The voltage signal iscompared to a reference voltage 2012 by a high-speed analog comparatorunit 2015, which asserts an event detection (ED) signal when the voltagesignal exceeds the reference voltage. The reference voltage 2012 may begenerated by a digital-to-analog converter (DAC) based on a referencevoltage value RVV supplied by the processing element 2020. Thecomparator unit 2015 may include a filter to help remove false positivesdue to noise. The ED signal may be provided to the processing element2020, and thence to the system controller block (SCB).

The PMB may also include an analog-to-digital converter (ADC) 2025 thatdigitizes the output voltage of the TIA 2010, and sends the resultingsample sequence to the processing element 2020 and/or the SCB, tosupport image reconstruction by the SCB (or image reconstruction by someagent external to the system 1800).

The processing element 2020 may receive the reference voltage value RVVfrom the SCB which, and forward it to the programmable reference voltage(PRV) block 2012. The PRV block may include a digital-to-analogconverter that creates the analog reference voltage used by thecomparator unit 2015.

In another embodiment of the PMB, the comparator function is performedin the digital domain, e.g., using a digital comparator unit DCU asshown in FIG. 21. The DCU takes as its inputs the sample sequencesupplied by the ADC 2025 and the digital reference voltage value RVVprovided by the processing element 2020.

The SCB may continuously monitor the photodiode monitoring blocks (PMBs)or a selected subset of the photodiode monitoring blocks, as shown atoperation 2210 of FIG. 22. For example, when the SCB determines that theselected subset of PMBs are all asserting their respective ED signals,the SCB may declare a spectral signature (SS) event, e.g., by assertingan SS signal.

Differential Image Comparison to Locate Spectral Event

In some embodiments, the system control block (SCB) may reconstruct apre-event image and an event-containing image in response to theassertion of the SS signal, as indicated at operation 2215 of FIG. 22.The pre-event image and the event-containing image may be reconstructedbased respectively on a pre-event block of samples and anevent-containing block of samples. The samples of each sample block maybe drawn (or derived) from the sample sequences corresponding to thespectral windows of interest. For example, the SCB may add the samplesequences corresponding to the spectral windows of interest to obtain asum sequence. The SCB may reconstruct the pre-event image based on ablock of samples from the sum sequence, i.e., a block of samples thatprecedes the occurrence of the event. Furthermore, the SCB mayreconstruct the event-containing image based on a second block ofsamples from the sum sequence, i.e., a block of samples that at leastpartially includes the temporal duration of the event. (The first andsecond block of samples may overlap in time.) At 2220, the SCB mayperform a differential image comparison between the pre-event image andevent-containing image to spatially identify the location of the sourceof the spectral event, e.g., as variously described above. At 2225, theSCB may reconstruct images based on corresponding blocks of sample dataderived from all the spectral windows. The images may be reconstructedperiodically, e.g., with programmable frequency. At 2230, the SCB maymerge a visual indication of the location of the spectral event into thefull-spectrum image(s) generated by operation 2225. The merged image(s)may be displayed via a display device.

In alternative embodiments, the SCB may continuously reconstruct imagesbased on sample data drawn from (or derived from) the sample sequencescorresponding to the spectral windows of interest (i.e., from theselected subset of PMBs), as indicated at 2215′ of FIG. 23. Thereconstruction process may involve summing the sample sequencescorresponding to the spectral windows of interest to obtain a sumsequence, and then generating images based on corresponding blocks(e.g., overlapping blocks) of samples from the sum sequence. The mostrecently reconstructed image may be continuously cached into memory. Atshown at 2220′ of FIG. 23, when the SS signal is asserted, the SCB may:(a) collect additional sample data from the sample sequencescorresponding to the spectral windows of interest (i.e., from the samplesequences provided by selected subset of PMBs) for a programmable amountof time; (b) reconstruct a new image based on a block of samplesincluding at least the additional sample data, e.g., as variouslydescribed above; and (c) perform a differential image comparison betweenthe cached image and new image to spatially identify the location of thesource of the spectral event. Again, a visual representation of thespatial location may be merged into the full-spectrum image(s) generatedby operation 2225.

Dual Path Architectures

In one set of embodiments, a system 2400 for spectral event detectionmay be configured as shown in FIG. 24. The system 2400 may include adigital micromirror device (DMD) 2405, a spectral separation subsystem2410, an array 2415 of light sensing elements, a sampling subsystem 2420and a detection unit 2425. (The system 2400 may also include any subsetof the features, element and embodiments described above in connectionwith system 100, system realization 200, system 1300, method 1700 andsystem 1800, and described below in connection with system 2800 andsystem 2900.)

The DMD 2405 may be configured to receive an incident light stream, andmodulate the incident light stream with a temporal sequence of spatialpatterns to obtain a modulated light stream MLS and a complementarymodulated light stream CMLS. The modulated light stream MLS comprisesportions of the incident light stream that are reflected at any giventime by micromirrors in a first of two orientation states. Thecomplementary modulated light stream CMLS comprises portions of theincident light stream that are reflected at any given time bymicromirrors in a second of the two orientation states. The DMD 2405 maybe realized as variously described above in connection with mirrors110M.

The spectral separation subsystem 2410 may be configured to receive themodulated light stream, and separate the modulated light stream into aplurality of wavelength components. As noted above, the plurality ofwavelength components may include a continuum of wavelength componentsand/or a set of discrete wavelength components. The spectral separationsubsystem 2410 may be configured as variously described above inconnection with spectral separation subsystem 1310.

The light sensing elements (e.g., photodiodes) of the array 2415 areconfigured to receive respective subsets (e.g., bands) of the wavelengthcomponents and to generate respective spectral element signals. Each ofthe spectral element signals represents intensity of the respectivesubset of the wavelength components as a function of time.

The sampling subsystem 2420 may be configured to sample the spectralelement signals in order to obtain respective spectrally-limited samplesequences. The spectrally-limited sample sequences may be stored in amemory of the system 2400.

The detection unit 2425 may be configured (e.g., as variously describedabove in connection with system 1300 and/or system 1800) to monitor aselected subset of the spectral element signals to detect an eventoccurring within a field of view corresponding to the incident lightstream. The action of detecting the event may include determining whenthe selected subset of the spectral element signals satisfy apre-determined signal condition.

In some embodiments, the system 2400 may also include an array 2515 oflight sensing elements, a sampling subsystem 2520 and a processing unit2525 as shown in FIG. 25.

The light sensing elements (e.g., photodiodes) of the array 2515 may beconfigured to convert respective spatial portions of the complementarymodulated light stream into respective spatial element signals, e.g., asvariously described in U.S. patent application Ser. No. 13/197,304. Alens may be used to image the complementary modulated light stream ontothe array 2515.

The sampling subsystem 2520 may be configured to sample the spatialelement signals to obtain respective spatially-limited sample sequences.The spatially-limited sample sequences may be stored into memory.

The processing unit 2525 may be configured to reconstruct a temporalsequence of images based on the spatially-limited sample sequences. Forexample, at each frame time, the processing unit may generate an imageby reconstructing a plurality of sub-images (i.e., one sub-image foreach of the light sensing elements), and concatenating the sub-images,i.e., joining the sub-images together along their boundaries. Eachsub-image represents a corresponding portion of the field of view, i.e.,the portion that is captured by the respective light sensing element.Each sub-image is reconstructed based on a current block of samples fromthe corresponding spatially-limited sample sequence. The processing unit2525 may be realized, e.g., as variously described above in connectionwith processing unit 150, processing unit 1330 and system controllerblock SCB.

In some embodiments, the system 2400 may also include a light sensingdevice 2615, an analog-to-digital converter (ADC) 2620 and a processingunit 2625 as shown in FIG. 26. The light sensing device 2615 may beconfigured to convert the complementary modulated light stream into adevice output signal representing intensity of the complementarymodulated light stream as a function of time. The light sensing device2615 may be realized as variously described above in connection withlight sensing device 130. (A lens may be used to focus or direct orconcentrate the complementary modulated light stream onto the lightsensing device.) The analog-to-digital converter (ADC) 2620 may beconfigured to sample the device output signal to obtain an output samplesequence. The output sample sequence may be stored into memory. Theprocessing unit 2625 may be configured to reconstruct a temporalsequence (e.g., a video sequence) of images based on the output samplesequence, e.g, as variously described above in connection with system100 and system realization 200. The temporal sequence of images may bedisplayed via a display device.

In some embodiments, system 2400 may include a dual TIR prism 2710 and alight sensing unit 2712 as shown in FIG. 27. (TIR is an acronym forTotal Internal Reflection.) The dual TIR prism may receive the incidentlight stream from an input path 2715 and transmit the incident lightstream to the DMD 2405. The dual TIR prism is further configured toreceive the modulated light stream MLS and the complementary modulatedlight stream CMLS from the DMD, to totally internally reflect themodulated light stream MLS onto an output path 2720 leading to thespectral separation subsystem 2410, and to totally internally reflectthe complementary modulated light stream CMLS onto an output path 2725leading to a light sensing unit 2730. The light sensing unit may beconfigured to receive the complementary modulated light stream CMLS fromthe output path 2725, and to generate one or more output signalsrepresentative of the complementary modulated light stream. For example,light sensing unit 2730 may be realized by the light sensing array 2515or the light sensing device 2615 described above.

FIG. 28 illustrates an embodiment 2800 of system 2400. A camera lens2805 may receive and operate on the incident light stream L. Theincident light stream L then passes through the dual TIR prism 2810 andis supplied to the DMD 2815. The DMD 2815 modulates the incident lightstream as variously described above to obtain a modulated light streamMLS and a complementary modulated light stream CMLS. The dual TIR prismreceives the modulated light stream MLS and complementary modulatedlight stream CMLS, and reflects those streams onto output paths leadingrespectively to lens 2817 and lens 2845. Lens 2817 may image themodulated light stream onto the diffraction grating 2820. Thediffraction grating 2820 may diffract the modulated light stream into aplurality of wavelength components, as variously described above. Acurved mirror 2825 may be used to reflect the wavelength components ontoan array 2830 of light sensing elements. The light sensing elementsconvert respective subsets (e.g., respective bands) of the wavelengthcomponents into respective signals. The sampling subsystem 2825 samplesthe signals to generate respective sample sequences. The detection unit2837 and processing unit 2840 may operate as variously described above.The lens 2845 may focus, image or direct or concentrate thecomplementary modulated light stream CMLS onto the light sensing unit2850. The light sensing unit 2850 may be realized by the light sensingarray 2515, in which case the lens 2845 may image the CMLS onto thelight sensing array 2515. Alternatively, the light sensing unit 2850 maybe realized by the light sensing device 2615, in which case the lens2845 may focus or direct or concentrate the CMLS onto the light sensingdevice 2615. The light sensing unit 2850 converts the CMLS into one ormore signals as variously described above. The sampling subsystem 2852converts the one or more signals respectively into one or more samplesequences. The processing unit 2840 may reconstruct a sequence of imagesbased on the one or more sample sequences.

Separately Sensing Zeroth Order and First Order Diffraction Beams

In one set of embodiments, a system 2900 may be configured as shown inFIG. 29. The system 2900 includes a light modulation unit 2910, adiffraction unit 2920, an array 2930 of light sensing elements, asampling subsystem 2935 and a detection unit 2940. (Furthermore, system2900 may include any subset of the features, elements and embodimentsdescribed above in connection with system 100, system realization 200,system 1300, method 1700, system 1800, system 2400 and system 2800.)

The light modulation unit 2910 may be configured to receive an incidentlight stream L, and modulate the incident light stream L with a temporalsequence of spatial patterns to obtain a modulated light stream MLS. Thelight modulation unit 2910 may be realized as variously described abovein connection with system 100, system realization 200 and system 1300.

The diffraction unit 2920 may be configured to diffract the modulatedlight stream into a zeroth-order diffraction beam B₀ and a first-orderdiffraction beam B₁. (For a basic tutorial on the subject ofdiffraction, see the Wikipedia page on diffraction athttp://en.wikipedia.org/wiki/Diffraction_grating.) Thus, the wavelengthcomponents present in the modulated light stream MLS are spatiallyseparated (angularly spread out) in the beam B₁ but not spatiallyseparated in the beam B₀. The diffraction unit may be realized by adiffraction grating, e.g., a transmission grating.

The light sensing elements of the light sensing array (LSA) 2930 may beconfigured to receive respective subsets (e.g., bands) of the wavelengthcomponents of the first-order beam B₁ and to generate respectivespectral element signals. Each of the spectral element signalsrepresents intensity of the respective subset of the wavelengthcomponents as a function of time.

The sampling subsystem (SSS) 2935 may be configured to sample thespectral element signals in order to obtain respectivespectrally-limited sample sequences. The spectrally-limited samplesequences may be stored into a memory of the system 2900.

The detection unit (DU) 2940 may be configured (e.g., as variouslydescribed above in connection with system 1300 and method 1700) tomonitor a selected subset of the spectral element signals to detect anevent occurring within a field of view corresponding to the incidentlight stream. The action of detecting the event includes determiningwhen the selected subset of the spectral element signals satisfy apre-determined signal condition, e.g., as variously described above.

In some embodiments, system 2900 may also include an array of lightsensing elements 2945, a sampling subsystem (SSS) 2950 and a processingunit 2955 as shown in FIG. 30.

The light sensing elements of the light sensing array (LSA) 2945 may beconfigured to convert spatial portions of the zeroth-order beam B₀ intorespective spatial element signals. Each of the spatial element signalsrepresents intensity of the respective spatial portion as a function oftime.

The sampling subsystem (SSS) 2950 may be configured to sample thespatial element signals to obtain respective spatially-limited samplesequences. The spatially-limited sample sequences may be stored intomemory.

The processing unit 2955 may be configured to reconstruct a temporalsequence of images based on the spatially-limited sample sequences. Forexample, each of the spatially-limited sample sequences may be used toreconstruct a corresponding temporal sequence of subimagesrepresentative of a corresponding portion of the field of view.)

In some embodiments, the system 2900 may include a focusing lens 2922and/or an imaging lens 2943 as shown in FIG. 31. The focusing lens 2922directs (e.g, focuses) the beam B₀ to the imaging lens 2950, whichimages the beam B₀ onto the sensing array 2950. The focusing lens 2922also directs (e.g., focuses) the beam B_(i) to the sensing array 2930.

In some embodiments, the system 2900 may include a light sensing device2946 and a sampling subsystem 2951 as shown in FIG. 29. The lightsensing device 2946 may be configured to convert the zeroth-order beamB₀ into a device output signal representing intensity of thezeroth-order beam as a function of time. The analog-to-digital converter(ADC) 2951 may be configured to sample the device output signal toobtain an output sample sequence (and to store the output samplesequence). The processing unit 2955 may be configured to reconstruct atemporal sequence of images based on the output sample sequence, e.g.,as variously described above. Furthermore, the system of FIG. 32 mayinclude a lens to focus or direct the beam B₀ onto the light sensingdevice 2946 and/or to focus or direct the beam B₁ onto the light sensingarray 2930, e.g., as variously described above.

In some embodiments, the system 2900 may also include a TIR prism pair3300 as shown in FIG. 33. (In these embodiments, the light modulationunit 2910 may be a digital micromirror device.) The TIR prism pair maybe configured to receive an input light stream ILS (from the externalenvironment) at an external surface 3305, to partially transmit andpartially reflect the input light stream at an internal interface 3310in order to respectively generate a transmitted light stream TLS and areflected light stream RLS, to output the transmitted light stream TLSat an external surface 3315, and to output the reflected light stream atexternal surface 3316 onto an output path OPP. The incident light streamL received by the light modulation unit 2910 is the same as thetransmitted light stream as outputted onto the first output path. Themodulated light stream MLS produced by the light modulation unit 2910enters the TIR prism pair at the surface 3315, is totally internallyreflected at internal interface 3310 and outputted (at external surface3317) onto a path leading to the diffraction unit 2920.

In some embodiments, the internal interface 3310 of the TIR prism pairmay be configured so that the reflected light stream is spectrallydifferent from the transmitted light stream. For example, the interfacemay realize a spectral filter. In one embodiment, the reflected lightstream RLS may comprise visible light while the transmitted light streamTLS comprises SWIR light.

In some embodiments, the system 2900 may also include an array 2965 oflight sensing elements. The array 2965 may be configured to receive thereflected light stream RLS from the output path OPP, and to capture atemporal sequence of images representative of the reflected lightstream. For example, the array 2865 may be a CMOS sensor array. Becausethe reflected light stream RLS is a partial reflection of the inputlight stream ILS and does not experience the modulating action of thelight modulation unit 2910, the images of the temporal sequence directlyrepresent the external scene, without the need to execute acompressive-sensing reconstruction algorithm. The temporal sequence ofimages may be displayed via a display device.

In some embodiments, the above-described embodiment of system 2900,i.e., including the array 2965 may be used to realize a dual-bandimager. For example, the array 2965 may be capture visible light imageswhile the light sensing array 2930 captures SWIR light. For example, theinternal interface 3310 may be configured to so that the reflected lightstream comprises only (or mostly) visible light while the modulatedlight stream MLS, after total internal reflection at the internalinterface 3310 comprises only (or mostly) SWIR light. The diffractionunit 2910 then separates the modulated light stream MLS into wavelengthcomponents within the SWIR range. Image registration may be necessary tointegrate the images from the different bands.

Any of the various embodiments described herein may be combined to formcomposite embodiments. Furthermore, any of the various features,embodiments and elements described in U.S. Provisional Application No.61/502,153 may be combined with any of the various embodiments describedherein.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

What is claimed is:
 1. A system comprising: a spectral separationsubsystem configured to receive a modulated light stream, wherein themodulated light stream is generated by modulating an incident lightstream with a temporal sequence of spatial patterns, wherein theincident light stream carries images from an external environment,wherein the spatial patterns belong to a set of measurement vectors thatis incoherent with a second set of vectors in which the images aresparse, wherein the spectral separation subsystem is configured toseparate the modulated light stream into a plurality of wavelengthcomponents; an array of light sensing elements, wherein the lightsensing elements are configured to receive respective subsets of thewavelength components and to generate respective electrical signals,wherein each of the electrical signals represents intensity of therespective subset of the wavelength components as a function of time; asampling subsystem configured to sample the electrical signals in orderto obtain respective sample sequences; a detection unit configured tomonitor a selected subset of the electrical signals to detect an eventoccurring within a field of view corresponding to the incident lightstream, wherein said detecting the event includes determining when theselected subset of electrical signals satisfy a pre-determined signalcondition; a processing unit configured to reconstruct a temporalsequence of images based on the sample sequences; a display deviceconfigured to the display the temporal sequence of images to a user;wherein the processing unit is configured to: in response to detectingsaid event, reconstruct a first image and a second image, wherein firstimage is reconstructed based on a first window of samples taken from thesample sequences corresponding to the selected subset, wherein thesecond image is reconstructed based on a second window of samples takenfrom the sample sequences corresponding to the selected subset, whereinthe first window corresponds to a first time interval prior to theevent, wherein the second window corresponds to a second time intervalthat at least partially includes the event; determine spatiallocalization information based on a difference between the first imageand the second image, wherein the spatial localization informationindicates where the event has occurred in the field of view; and injecta visual representation of the spatial localization information into atleast a subset of the images of said temporal sequence of images,enabling the user to identify where the event has occurred in the fieldof view.
 2. The system of claim 1, wherein the pre-determined signalcondition is the condition that the electrical signals of the selectedsubset simultaneously exceed respective programmable thresholds.
 3. Thesystem of claim 1, wherein the detection unit includes a plurality ofanalog comparator circuits configured to compare respective electricalsignals of the selected subset to respective programmable thresholds. 4.The system of claim 1, wherein the detection unit includes a pluralityof digital comparator circuits, wherein each of the digital comparatorcircuits is configured to compare a respective one of the samplesequences of the selected subset to a respective programmable threshold.5. The system of claim 1, wherein the detection unit includes aplurality of digital circuits, wherein each of the digital circuits isconfigured to compare a rate of change of a respective one of the samplesequences of the selected subset to a respective programmable threshold.6. The system of claim 1, wherein the pre-determined signal condition isthe logical AND of a first condition and a second condition, wherein thefirst condition is that the electrical signals of the selected subsetsimultaneously exceed respective programmable value thresholds, whereinthe second condition is that rates of change of the respectiveelectrical signals of the selected subset simultaneously exceedrespective programmable rate thresholds.
 7. The system of claim 1,wherein the pre-determined signal condition is the condition that theelectrical signals of the selected subset have respective values thatconform to a pre-determined spectral signature.
 8. The system of claim1, wherein said plurality of wavelength components comprises one or morecontinuous distributions of wavelength components spanning a wavelengthrange.
 9. The system of claim 1, wherein the array of light sensingelements comprises a linear array.
 10. The system of claim 1, whereinthe processing unit is configured to: in response to detecting saidevent, perform a search process on one or more of the sample sequencesbelonging to the selected subset and during the occurrence of the eventin order to identify a spatial subregion within the field of view thatcontains the event, wherein the search process includes: injectingsearch patterns into the temporal sequence of spatial patterns; andanalyzing the samples of the one or more sample sequences in response tothe injection of the search patterns.
 11. The system of claim 10,wherein the search process includes a hierarchical search based on aquadtree, wherein the quadtree corresponds to a recursive partitioningof the field of view into rectangular subsets.
 12. The system of claim10, wherein the processing unit is further configured to: inject avisual representation of the spatial subregion into at least a subset ofthe images of said temporal sequence of images.
 13. The system of claim1, wherein the selected subset of the electrical signals isprogrammable.
 14. A method comprising: receiving a modulated lightstream, wherein the modulated light stream is generated by modulating anincident light stream with a temporal sequence of spatial patterns;separating the modulated light stream into a plurality of wavelengthcomponents; converting subsets of the wavelength components intorespective electrical signals, wherein each of the electrical signalsrepresents intensity of the respective subset of the wavelengthcomponents as a function of time; sampling the electrical signals inorder to obtain respective sample sequences; monitoring a selectedsubset of the electrical signals to detect an event occurring within afield of view corresponding to the incident light stream, wherein saiddetecting the event includes determining when the selected subset ofelectrical signals satisfy a pre-determined signal condition; andreconstructing a temporal sequence of images based on the samplesequences; displaying the sequence of temporal images via a displaydevice; in response to detecting said event, reconstructing a firstimage and a second image, wherein first image is reconstructed based ona first window of samples taken from the sample sequences correspondingto the selected subset, wherein the second image is reconstructed basedon a second window of samples taken from the sample sequencescorresponding to the selected subset, wherein the first windowcorresponds to a first time interval prior to the event, wherein thesecond window corresponds to a second time interval that at leastpartially includes the event; determining spatial localizationinformation based on a difference between the first image and the secondimage, wherein the spatial localization information indicates where theevent has occurred in the field of view; and injecting a visualrepresentation of the spatial localization information into a subset ofthe images of said temporal sequence of images, enabling a user toidentify where the event has occurred in the field of view.
 15. Themethod of claim 14, wherein the pre-determined signal condition is thecondition that the electrical signals of the selected subsetsimultaneously exceed respective programmable thresholds.
 16. The methodof claim 14, wherein the pre-determined signal condition is thecondition that rates of change of the respective electrical signals ofthe selected subset exceed respective programmable thresholds.
 17. Themethod of claim 14, wherein the pre-determined signal condition is thecondition that the electrical signals of the selected subset haverespective values that conform to a pre-determined spectral signature.18. The method of claim 14, wherein said reconstructing the temporalsequence of images is initiated in response to said detection of theevent.
 19. The method of claim 14, further comprising: in response todetecting said event, performing a search process on one or more of thesample sequences belonging to the selected subset and during theoccurrence of the event in order to identify a spatial subregion of thefield of view that contains the event, wherein the search processincludes: injecting search patterns into the temporal sequence ofspatial patterns; and analyzing the samples of the one or more samplesequences in response to the injection of the search patterns.
 20. Themethod of claim 19, wherein the search process includes a hierarchicalsearch based on a quadtree, wherein the quadtree corresponds to arecursive partitioning of the field of view into rectangular subsets.21. The method of claim 19, further comprising: injecting a visualrepresentation of the spatial subregion into at least a subset of theimages of said temporal sequence of images.
 22. The method of claim 14,wherein the selected subset of the electrical signals is programmable.